Note: This is test shard 2 of 8.
[==========] Running 8 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from RebalanceIgnoredTserversTest
[ RUN ] RebalanceIgnoredTserversTest.Basic
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 20:45:24.563652 30849 test_util.cc:268] Using random seed: -1363985746
I20240502 20:45:24.570875 30849 ts_itest-base.cc:112] Starting cluster with:
I20240502 20:45:24.570993 30849 ts_itest-base.cc:113] --------------
I20240502 20:45:24.571033 30849 ts_itest-base.cc:114] 5 tablet servers
I20240502 20:45:24.571066 30849 ts_itest-base.cc:115] 3 replicas per TS
I20240502 20:45:24.571084 30849 ts_itest-base.cc:116] --------------
2024-05-02T20:45:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T20:45:24Z Disabled control of system clock
I20240502 20:45:24.581369 30849 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskolF7xA/build/debug/bin/kudu
/tmp/dist-test-taskolF7xA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.32.126:34843
--webserver_interface=127.30.32.126
--webserver_port=0
--builtin_ntp_servers=127.30.32.84:41473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.32.126:34843
--enable_range_replica_placement=false with env {}
W20240502 20:45:24.688777 30861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:24.689047 30861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:24.689131 30861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:24.692128 30861 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 20:45:24.692185 30861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:24.692205 30861 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 20:45:24.692222 30861 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 20:45:24.695935 30861 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.32.84:41473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.30.32.126:34843
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.32.126:34843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.30.32.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.30861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:24.696851 30861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:24.697990 30861 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:24.705461 30867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:24.705619 30866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:24.706202 30869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:24.706790 30861 server_base.cc:1034] running on GCE node
I20240502 20:45:24.707273 30861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:24.708170 30861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:24.709307 30861 hybrid_clock.cc:648] HybridClock initialized: now 1714682724709288 us; error 36 us; skew 500 ppm
I20240502 20:45:24.717152 30861 webserver.cc:460] Webserver started at http://127.30.32.126:43409/ using document root <none> and password file <none>
I20240502 20:45:24.717715 30861 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:24.717777 30861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:24.717945 30861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:24.719614 30861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "3acaa4a4bfc949ef9f923799c2cc36e9"
format_stamp: "Formatted at 2024-05-02 20:45:24 on dist-test-slave-6336"
I20240502 20:45:24.720069 30861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "3acaa4a4bfc949ef9f923799c2cc36e9"
format_stamp: "Formatted at 2024-05-02 20:45:24 on dist-test-slave-6336"
I20240502 20:45:24.723510 30861 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20240502 20:45:24.725615 30875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:24.726572 30861 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 20:45:24.726681 30861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/wal
uuid: "3acaa4a4bfc949ef9f923799c2cc36e9"
format_stamp: "Formatted at 2024-05-02 20:45:24 on dist-test-slave-6336"
I20240502 20:45:24.726763 30861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:24.750873 30861 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:24.751549 30861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:24.751711 30861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:24.758080 30861 rpc_server.cc:307] RPC server started. Bound to: 127.30.32.126:34843
I20240502 20:45:24.758085 30927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.32.126:34843 every 8 connection(s)
I20240502 20:45:24.759008 30861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 20:45:24.761495 30928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:24.766287 30928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3acaa4a4bfc949ef9f923799c2cc36e9: Bootstrap starting.
I20240502 20:45:24.767453 30849 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskolF7xA/build/debug/bin/kudu as pid 30861
I20240502 20:45:24.767603 30849 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 20:45:24.768894 30928 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3acaa4a4bfc949ef9f923799c2cc36e9: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:24.769737 30928 log.cc:824] T 00000000000000000000000000000000 P 3acaa4a4bfc949ef9f923799c2cc36e9: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:24.771556 30928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3acaa4a4bfc949ef9f923799c2cc36e9: No bootstrap required, opened a new log
I20240502 20:45:24.775333 30928 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3acaa4a4bfc949ef9f923799c2cc36e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3acaa4a4bfc949ef9f923799c2cc36e9" member_type: VOTER last_known_addr { host: "127.30.32.126" port: 34843 } }
I20240502 20:45:24.775530 30928 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3acaa4a4bfc949ef9f923799c2cc36e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:24.775646 30928 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3acaa4a4bfc949ef9f923799c2cc36e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3acaa4a4bfc949ef9f923799c2cc36e9, State: Initialized, Role: FOLLOWER
I20240502 20:45:24.776122 30928 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3acaa4a4bfc949ef9f923799c2cc36e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3acaa4a4bfc949ef9f923799c2cc36e9" member_type: VOTER last_known_addr { host: "127.30.32.126" port: 34843 } }
I20240502 20:45:24.776300 30928 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3acaa4a4bfc949ef9f923799c2cc36e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 20:45:24.776374 30928 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3acaa4a4bfc949ef9f923799c2cc36e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 20:45:24.776474 30928 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3acaa4a4bfc949ef9f923799c2cc36e9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:24.777305 30928 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3acaa4a4bfc949ef9f923799c2cc36e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3acaa4a4bfc949ef9f923799c2cc36e9" member_type: VOTER last_known_addr { host: "127.30.32.126" port: 34843 } }
I20240502 20:45:24.777611 30928 leader_election.cc:302] T 00000000000000000000000000000000 P 3acaa4a4bfc949ef9f923799c2cc36e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3acaa4a4bfc949ef9f923799c2cc36e9; no voters:
I20240502 20:45:24.777925 30928 leader_election.cc:288] T 00000000000000000000000000000000 P 3acaa4a4bfc949ef9f923799c2cc36e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 20:45:24.778216 30933 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3acaa4a4bfc949ef9f923799c2cc36e9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:24.778457 30933 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3acaa4a4bfc949ef9f923799c2cc36e9 [term 1 LEADER]: Becoming Leader. State: Replica: 3acaa4a4bfc949ef9f923799c2cc36e9, State: Running, Role: LEADER
I20240502 20:45:24.778801 30933 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3acaa4a4bfc949ef9f923799c2cc36e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3acaa4a4bfc949ef9f923799c2cc36e9" member_type: VOTER last_known_addr { host: "127.30.32.126" port: 34843 } }
I20240502 20:45:24.779119 30928 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3acaa4a4bfc949ef9f923799c2cc36e9 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 20:45:24.780279 30934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3acaa4a4bfc949ef9f923799c2cc36e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3acaa4a4bfc949ef9f923799c2cc36e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3acaa4a4bfc949ef9f923799c2cc36e9" member_type: VOTER last_known_addr { host: "127.30.32.126" port: 34843 } } }
I20240502 20:45:24.780405 30934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3acaa4a4bfc949ef9f923799c2cc36e9 [sys.catalog]: This master's current role is: LEADER
I20240502 20:45:24.780800 30935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3acaa4a4bfc949ef9f923799c2cc36e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3acaa4a4bfc949ef9f923799c2cc36e9. Latest consensus state: current_term: 1 leader_uuid: "3acaa4a4bfc949ef9f923799c2cc36e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3acaa4a4bfc949ef9f923799c2cc36e9" member_type: VOTER last_known_addr { host: "127.30.32.126" port: 34843 } } }
I20240502 20:45:24.780942 30935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3acaa4a4bfc949ef9f923799c2cc36e9 [sys.catalog]: This master's current role is: LEADER
W20240502 20:45:24.783872 30948 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3acaa4a4bfc949ef9f923799c2cc36e9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 20:45:24.783948 30948 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 20:45:24.784022 30942 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 20:45:24.784889 30942 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 20:45:24.789155 30942 catalog_manager.cc:1347] Generated new cluster ID: a369f4b187dc40acb275e9f7fa182a48
I20240502 20:45:24.789251 30942 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 20:45:24.797480 30942 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 20:45:24.798612 30942 catalog_manager.cc:1504] Loading token signing keys...
I20240502 20:45:24.808403 30942 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 3acaa4a4bfc949ef9f923799c2cc36e9: Generated new TSK 0
I20240502 20:45:24.809262 30942 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 20:45:24.814108 30849 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskolF7xA/build/debug/bin/kudu
/tmp/dist-test-taskolF7xA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.32.65:0
--local_ip_for_outbound_sockets=127.30.32.65
--webserver_interface=127.30.32.65
--webserver_port=0
--tserver_master_addrs=127.30.32.126:34843
--builtin_ntp_servers=127.30.32.84:41473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20240502 20:45:24.921451 30952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:24.921653 30952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:24.921703 30952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:24.924674 30952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:24.924782 30952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.32.65
I20240502 20:45:24.928301 30952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.32.84:41473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.32.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.30.32.65
--webserver_port=0
--tserver_master_addrs=127.30.32.126:34843
--never_fsync=true
--heap_profile_path=/tmp/kudu.30952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.32.65
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:24.929176 30952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:24.930327 30952 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:24.938781 30960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:24.939003 30957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:24.939561 30958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:24.940164 30952 server_base.cc:1034] running on GCE node
I20240502 20:45:24.940513 30952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:24.941062 30952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:24.942207 30952 hybrid_clock.cc:648] HybridClock initialized: now 1714682724942188 us; error 31 us; skew 500 ppm
I20240502 20:45:24.943914 30952 webserver.cc:460] Webserver started at http://127.30.32.65:34415/ using document root <none> and password file <none>
I20240502 20:45:24.944432 30952 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:24.944491 30952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:24.944662 30952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:24.946288 30952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "49d17dbb3e5647f5998cdac93f6011ae"
format_stamp: "Formatted at 2024-05-02 20:45:24 on dist-test-slave-6336"
I20240502 20:45:24.946733 30952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "49d17dbb3e5647f5998cdac93f6011ae"
format_stamp: "Formatted at 2024-05-02 20:45:24 on dist-test-slave-6336"
I20240502 20:45:24.949870 30952 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20240502 20:45:24.951838 30966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:24.952733 30952 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 20:45:24.952842 30952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "49d17dbb3e5647f5998cdac93f6011ae"
format_stamp: "Formatted at 2024-05-02 20:45:24 on dist-test-slave-6336"
I20240502 20:45:24.952915 30952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:24.964743 30952 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:24.965337 30952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:24.965482 30952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:24.965960 30952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:24.966773 30952 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:24.966820 30952 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:24.966876 30952 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:24.966902 30952 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:24.975237 30952 rpc_server.cc:307] RPC server started. Bound to: 127.30.32.65:39209
I20240502 20:45:24.975270 31079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.32.65:39209 every 8 connection(s)
I20240502 20:45:24.976083 30952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 20:45:24.979629 30849 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskolF7xA/build/debug/bin/kudu as pid 30952
I20240502 20:45:24.979724 30849 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 20:45:24.981992 30849 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskolF7xA/build/debug/bin/kudu
/tmp/dist-test-taskolF7xA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.32.66:0
--local_ip_for_outbound_sockets=127.30.32.66
--webserver_interface=127.30.32.66
--webserver_port=0
--tserver_master_addrs=127.30.32.126:34843
--builtin_ntp_servers=127.30.32.84:41473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240502 20:45:24.984902 31080 heartbeater.cc:347] Connected to a master server at 127.30.32.126:34843
I20240502 20:45:24.985162 31080 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:24.985816 31080 heartbeater.cc:511] Master 127.30.32.126:34843 requested a full tablet report, sending...
I20240502 20:45:24.987191 30892 ts_manager.cc:194] Registered new tserver with Master: 49d17dbb3e5647f5998cdac93f6011ae (127.30.32.65:39209)
I20240502 20:45:24.988276 30892 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.32.65:56495
W20240502 20:45:25.089076 31083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:25.089286 31083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:25.089330 31083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:25.092391 31083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:25.092500 31083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.32.66
I20240502 20:45:25.096143 31083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.32.84:41473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.32.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.30.32.66
--webserver_port=0
--tserver_master_addrs=127.30.32.126:34843
--never_fsync=true
--heap_profile_path=/tmp/kudu.31083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.32.66
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:25.097111 31083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:25.098291 31083 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:25.106264 31088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:25.106874 31091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:25.107429 31089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:25.108008 31083 server_base.cc:1034] running on GCE node
I20240502 20:45:25.108366 31083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:25.108906 31083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:25.110050 31083 hybrid_clock.cc:648] HybridClock initialized: now 1714682725110029 us; error 38 us; skew 500 ppm
I20240502 20:45:25.111768 31083 webserver.cc:460] Webserver started at http://127.30.32.66:45117/ using document root <none> and password file <none>
I20240502 20:45:25.112298 31083 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:25.112357 31083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:25.112529 31083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:25.114194 31083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "77022bea76324055a55a1d7134cf4809"
format_stamp: "Formatted at 2024-05-02 20:45:25 on dist-test-slave-6336"
I20240502 20:45:25.114640 31083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "77022bea76324055a55a1d7134cf4809"
format_stamp: "Formatted at 2024-05-02 20:45:25 on dist-test-slave-6336"
I20240502 20:45:25.117902 31083 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 20:45:25.119983 31097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:25.120941 31083 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:25.121070 31083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "77022bea76324055a55a1d7134cf4809"
format_stamp: "Formatted at 2024-05-02 20:45:25 on dist-test-slave-6336"
I20240502 20:45:25.121155 31083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:25.137542 31083 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:25.138231 31083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:25.138391 31083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:25.138948 31083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:25.139868 31083 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:25.139914 31083 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:25.139973 31083 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:25.140000 31083 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:25.148288 31083 rpc_server.cc:307] RPC server started. Bound to: 127.30.32.66:34315
I20240502 20:45:25.148315 31210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.32.66:34315 every 8 connection(s)
I20240502 20:45:25.149271 31083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 20:45:25.157467 31211 heartbeater.cc:347] Connected to a master server at 127.30.32.126:34843
I20240502 20:45:25.157577 30849 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskolF7xA/build/debug/bin/kudu as pid 31083
I20240502 20:45:25.157687 30849 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 20:45:25.157686 31211 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:25.158200 31211 heartbeater.cc:511] Master 127.30.32.126:34843 requested a full tablet report, sending...
I20240502 20:45:25.159178 30892 ts_manager.cc:194] Registered new tserver with Master: 77022bea76324055a55a1d7134cf4809 (127.30.32.66:34315)
I20240502 20:45:25.159885 30849 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskolF7xA/build/debug/bin/kudu
/tmp/dist-test-taskolF7xA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.32.67:0
--local_ip_for_outbound_sockets=127.30.32.67
--webserver_interface=127.30.32.67
--webserver_port=0
--tserver_master_addrs=127.30.32.126:34843
--builtin_ntp_servers=127.30.32.84:41473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240502 20:45:25.159907 30892 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.32.66:52355
W20240502 20:45:25.267850 31214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:25.268059 31214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:25.268114 31214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:25.271189 31214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:25.271303 31214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.32.67
I20240502 20:45:25.274843 31214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.32.84:41473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.32.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.30.32.67
--webserver_port=0
--tserver_master_addrs=127.30.32.126:34843
--never_fsync=true
--heap_profile_path=/tmp/kudu.31214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.32.67
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:25.275664 31214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:25.276734 31214 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:25.285132 31219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:25.285573 31222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:25.285790 31220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:25.286412 31214 server_base.cc:1034] running on GCE node
I20240502 20:45:25.286799 31214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:25.287360 31214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:25.288512 31214 hybrid_clock.cc:648] HybridClock initialized: now 1714682725288495 us; error 29 us; skew 500 ppm
I20240502 20:45:25.290333 31214 webserver.cc:460] Webserver started at http://127.30.32.67:32983/ using document root <none> and password file <none>
I20240502 20:45:25.290890 31214 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:25.290951 31214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:25.291143 31214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:25.292899 31214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "7f786b85fad644d1be83908499a0c175"
format_stamp: "Formatted at 2024-05-02 20:45:25 on dist-test-slave-6336"
I20240502 20:45:25.293424 31214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "7f786b85fad644d1be83908499a0c175"
format_stamp: "Formatted at 2024-05-02 20:45:25 on dist-test-slave-6336"
I20240502 20:45:25.296710 31214 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240502 20:45:25.298872 31228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:25.299829 31214 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:25.299933 31214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "7f786b85fad644d1be83908499a0c175"
format_stamp: "Formatted at 2024-05-02 20:45:25 on dist-test-slave-6336"
I20240502 20:45:25.300014 31214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:25.312021 31214 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:25.312630 31214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:25.312772 31214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:25.313303 31214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:25.314152 31214 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:25.314198 31214 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:25.314260 31214 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:25.314288 31214 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:25.322609 31214 rpc_server.cc:307] RPC server started. Bound to: 127.30.32.67:34651
I20240502 20:45:25.322633 31341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.32.67:34651 every 8 connection(s)
I20240502 20:45:25.323551 31214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 20:45:25.325577 30849 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskolF7xA/build/debug/bin/kudu as pid 31214
I20240502 20:45:25.325670 30849 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 20:45:25.327801 30849 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskolF7xA/build/debug/bin/kudu
/tmp/dist-test-taskolF7xA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.32.68:0
--local_ip_for_outbound_sockets=127.30.32.68
--webserver_interface=127.30.32.68
--webserver_port=0
--tserver_master_addrs=127.30.32.126:34843
--builtin_ntp_servers=127.30.32.84:41473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240502 20:45:25.333259 31342 heartbeater.cc:347] Connected to a master server at 127.30.32.126:34843
I20240502 20:45:25.333470 31342 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:25.333971 31342 heartbeater.cc:511] Master 127.30.32.126:34843 requested a full tablet report, sending...
I20240502 20:45:25.334956 30892 ts_manager.cc:194] Registered new tserver with Master: 7f786b85fad644d1be83908499a0c175 (127.30.32.67:34651)
I20240502 20:45:25.335774 30892 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.32.67:43399
W20240502 20:45:25.434134 31345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:25.434345 31345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:25.434386 31345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:25.437397 31345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:25.437496 31345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.32.68
I20240502 20:45:25.441013 31345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.32.84:41473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.32.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.30.32.68
--webserver_port=0
--tserver_master_addrs=127.30.32.126:34843
--never_fsync=true
--heap_profile_path=/tmp/kudu.31345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.32.68
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:25.441856 31345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:25.442926 31345 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:25.451020 31351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:25.451668 31353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:25.452235 31350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:25.452854 31345 server_base.cc:1034] running on GCE node
I20240502 20:45:25.453226 31345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:25.453788 31345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:25.454943 31345 hybrid_clock.cc:648] HybridClock initialized: now 1714682725454921 us; error 35 us; skew 500 ppm
I20240502 20:45:25.456815 31345 webserver.cc:460] Webserver started at http://127.30.32.68:41555/ using document root <none> and password file <none>
I20240502 20:45:25.457422 31345 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:25.457479 31345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:25.457670 31345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:25.459347 31345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "eba1ba06d3dd42279caed807795fb8ed"
format_stamp: "Formatted at 2024-05-02 20:45:25 on dist-test-slave-6336"
I20240502 20:45:25.459810 31345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "eba1ba06d3dd42279caed807795fb8ed"
format_stamp: "Formatted at 2024-05-02 20:45:25 on dist-test-slave-6336"
I20240502 20:45:25.463181 31345 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 20:45:25.465247 31359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:25.466184 31345 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 20:45:25.466310 31345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "eba1ba06d3dd42279caed807795fb8ed"
format_stamp: "Formatted at 2024-05-02 20:45:25 on dist-test-slave-6336"
I20240502 20:45:25.466388 31345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:25.479688 31345 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:25.480448 31345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:25.480618 31345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:25.481242 31345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:25.482208 31345 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:25.482259 31345 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:25.482302 31345 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:25.482333 31345 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:25.490633 31345 rpc_server.cc:307] RPC server started. Bound to: 127.30.32.68:35973
I20240502 20:45:25.490664 31472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.32.68:35973 every 8 connection(s)
I20240502 20:45:25.491569 31345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 20:45:25.493297 30849 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskolF7xA/build/debug/bin/kudu as pid 31345
I20240502 20:45:25.493408 30849 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 20:45:25.495469 30849 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskolF7xA/build/debug/bin/kudu
/tmp/dist-test-taskolF7xA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.32.69:0
--local_ip_for_outbound_sockets=127.30.32.69
--webserver_interface=127.30.32.69
--webserver_port=0
--tserver_master_addrs=127.30.32.126:34843
--builtin_ntp_servers=127.30.32.84:41473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240502 20:45:25.500772 31473 heartbeater.cc:347] Connected to a master server at 127.30.32.126:34843
I20240502 20:45:25.501075 31473 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:25.501809 31473 heartbeater.cc:511] Master 127.30.32.126:34843 requested a full tablet report, sending...
I20240502 20:45:25.502926 30892 ts_manager.cc:194] Registered new tserver with Master: eba1ba06d3dd42279caed807795fb8ed (127.30.32.68:35973)
I20240502 20:45:25.503580 30892 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.32.68:49515
W20240502 20:45:25.606477 31476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:25.606683 31476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:25.606745 31476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:25.609818 31476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:25.609949 31476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.32.69
I20240502 20:45:25.613569 31476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.32.84:41473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.32.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.30.32.69
--webserver_port=0
--tserver_master_addrs=127.30.32.126:34843
--never_fsync=true
--heap_profile_path=/tmp/kudu.31476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.32.69
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:25.614439 31476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:25.615531 31476 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:25.623456 31482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:25.624087 31481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:25.624711 31484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:25.625348 31476 server_base.cc:1034] running on GCE node
I20240502 20:45:25.625700 31476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:25.626214 31476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:25.627373 31476 hybrid_clock.cc:648] HybridClock initialized: now 1714682725627353 us; error 35 us; skew 500 ppm
I20240502 20:45:25.629168 31476 webserver.cc:460] Webserver started at http://127.30.32.69:43545/ using document root <none> and password file <none>
I20240502 20:45:25.629705 31476 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:25.629773 31476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:25.629959 31476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:25.631779 31476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "0e3a7d1630044df7ae01d3f144fa4fe3"
format_stamp: "Formatted at 2024-05-02 20:45:25 on dist-test-slave-6336"
I20240502 20:45:25.635459 31476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "0e3a7d1630044df7ae01d3f144fa4fe3"
format_stamp: "Formatted at 2024-05-02 20:45:25 on dist-test-slave-6336"
I20240502 20:45:25.639030 31476 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 20:45:25.641397 31490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:25.642401 31476 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 20:45:25.642517 31476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "0e3a7d1630044df7ae01d3f144fa4fe3"
format_stamp: "Formatted at 2024-05-02 20:45:25 on dist-test-slave-6336"
I20240502 20:45:25.642606 31476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:25.663465 31476 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:25.664220 31476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:25.664391 31476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:25.664963 31476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:25.665930 31476 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:25.665979 31476 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:25.666038 31476 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:25.666064 31476 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:25.675282 31476 rpc_server.cc:307] RPC server started. Bound to: 127.30.32.69:44221
I20240502 20:45:25.675307 31604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.32.69:44221 every 8 connection(s)
I20240502 20:45:25.676218 31476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 20:45:25.681541 30849 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskolF7xA/build/debug/bin/kudu as pid 31476
I20240502 20:45:25.681638 30849 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 20:45:25.684787 31605 heartbeater.cc:347] Connected to a master server at 127.30.32.126:34843
I20240502 20:45:25.685017 31605 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:25.685518 31605 heartbeater.cc:511] Master 127.30.32.126:34843 requested a full tablet report, sending...
I20240502 20:45:25.686378 30892 ts_manager.cc:194] Registered new tserver with Master: 0e3a7d1630044df7ae01d3f144fa4fe3 (127.30.32.69:44221)
I20240502 20:45:25.686910 30892 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.32.69:48273
I20240502 20:45:25.695684 30849 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 20:45:25.707922 30892 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35940:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 20:45:25.725364 31145 tablet_service.cc:1467] Processing CreateTablet for tablet aec87a906eaf4295a11cdb346fa7421e (DEFAULT_TABLE table=TestTable [id=e8edd2bfffae40a680bfa0a07d381acc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:25.725390 31276 tablet_service.cc:1467] Processing CreateTablet for tablet aec87a906eaf4295a11cdb346fa7421e (DEFAULT_TABLE table=TestTable [id=e8edd2bfffae40a680bfa0a07d381acc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:25.725395 31014 tablet_service.cc:1467] Processing CreateTablet for tablet aec87a906eaf4295a11cdb346fa7421e (DEFAULT_TABLE table=TestTable [id=e8edd2bfffae40a680bfa0a07d381acc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:25.726250 31145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aec87a906eaf4295a11cdb346fa7421e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:25.726264 31014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aec87a906eaf4295a11cdb346fa7421e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:25.726274 31276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aec87a906eaf4295a11cdb346fa7421e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:25.731473 31622 tablet_bootstrap.cc:492] T aec87a906eaf4295a11cdb346fa7421e P 77022bea76324055a55a1d7134cf4809: Bootstrap starting.
I20240502 20:45:25.732515 31624 tablet_bootstrap.cc:492] T aec87a906eaf4295a11cdb346fa7421e P 7f786b85fad644d1be83908499a0c175: Bootstrap starting.
I20240502 20:45:25.732555 31623 tablet_bootstrap.cc:492] T aec87a906eaf4295a11cdb346fa7421e P 49d17dbb3e5647f5998cdac93f6011ae: Bootstrap starting.
I20240502 20:45:25.733969 31622 tablet_bootstrap.cc:654] T aec87a906eaf4295a11cdb346fa7421e P 77022bea76324055a55a1d7134cf4809: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:25.734644 31623 tablet_bootstrap.cc:654] T aec87a906eaf4295a11cdb346fa7421e P 49d17dbb3e5647f5998cdac93f6011ae: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:25.734644 31624 tablet_bootstrap.cc:654] T aec87a906eaf4295a11cdb346fa7421e P 7f786b85fad644d1be83908499a0c175: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:25.734632 31622 log.cc:824] T aec87a906eaf4295a11cdb346fa7421e P 77022bea76324055a55a1d7134cf4809: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:25.735423 31623 log.cc:824] T aec87a906eaf4295a11cdb346fa7421e P 49d17dbb3e5647f5998cdac93f6011ae: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:25.735423 31624 log.cc:824] T aec87a906eaf4295a11cdb346fa7421e P 7f786b85fad644d1be83908499a0c175: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:25.736250 31622 tablet_bootstrap.cc:492] T aec87a906eaf4295a11cdb346fa7421e P 77022bea76324055a55a1d7134cf4809: No bootstrap required, opened a new log
I20240502 20:45:25.736583 31622 ts_tablet_manager.cc:1367] T aec87a906eaf4295a11cdb346fa7421e P 77022bea76324055a55a1d7134cf4809: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240502 20:45:25.737048 31623 tablet_bootstrap.cc:492] T aec87a906eaf4295a11cdb346fa7421e P 49d17dbb3e5647f5998cdac93f6011ae: No bootstrap required, opened a new log
I20240502 20:45:25.737048 31624 tablet_bootstrap.cc:492] T aec87a906eaf4295a11cdb346fa7421e P 7f786b85fad644d1be83908499a0c175: No bootstrap required, opened a new log
I20240502 20:45:25.737216 31624 ts_tablet_manager.cc:1367] T aec87a906eaf4295a11cdb346fa7421e P 7f786b85fad644d1be83908499a0c175: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 20:45:25.737216 31623 ts_tablet_manager.cc:1367] T aec87a906eaf4295a11cdb346fa7421e P 49d17dbb3e5647f5998cdac93f6011ae: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 20:45:25.739341 31622 raft_consensus.cc:348] T aec87a906eaf4295a11cdb346fa7421e P 77022bea76324055a55a1d7134cf4809 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f786b85fad644d1be83908499a0c175" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 34651 } } peers { permanent_uuid: "49d17dbb3e5647f5998cdac93f6011ae" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 39209 } } peers { permanent_uuid: "77022bea76324055a55a1d7134cf4809" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 34315 } }
I20240502 20:45:25.739533 31622 raft_consensus.cc:374] T aec87a906eaf4295a11cdb346fa7421e P 77022bea76324055a55a1d7134cf4809 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:25.739589 31622 raft_consensus.cc:729] T aec87a906eaf4295a11cdb346fa7421e P 77022bea76324055a55a1d7134cf4809 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77022bea76324055a55a1d7134cf4809, State: Initialized, Role: FOLLOWER
I20240502 20:45:25.739996 31622 consensus_queue.cc:260] T aec87a906eaf4295a11cdb346fa7421e P 77022bea76324055a55a1d7134cf4809 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f786b85fad644d1be83908499a0c175" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 34651 } } peers { permanent_uuid: "49d17dbb3e5647f5998cdac93f6011ae" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 39209 } } peers { permanent_uuid: "77022bea76324055a55a1d7134cf4809" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 34315 } }
I20240502 20:45:25.740453 31623 raft_consensus.cc:348] T aec87a906eaf4295a11cdb346fa7421e P 49d17dbb3e5647f5998cdac93f6011ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f786b85fad644d1be83908499a0c175" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 34651 } } peers { permanent_uuid: "49d17dbb3e5647f5998cdac93f6011ae" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 39209 } } peers { permanent_uuid: "77022bea76324055a55a1d7134cf4809" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 34315 } }
I20240502 20:45:25.740725 31623 raft_consensus.cc:374] T aec87a906eaf4295a11cdb346fa7421e P 49d17dbb3e5647f5998cdac93f6011ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:25.740797 31624 raft_consensus.cc:348] T aec87a906eaf4295a11cdb346fa7421e P 7f786b85fad644d1be83908499a0c175 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f786b85fad644d1be83908499a0c175" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 34651 } } peers { permanent_uuid: "49d17dbb3e5647f5998cdac93f6011ae" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 39209 } } peers { permanent_uuid: "77022bea76324055a55a1d7134cf4809" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 34315 } }
I20240502 20:45:25.740890 31622 ts_tablet_manager.cc:1398] T aec87a906eaf4295a11cdb346fa7421e P 77022bea76324055a55a1d7134cf4809: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240502 20:45:25.740887 31211 heartbeater.cc:503] Master 127.30.32.126:34843 was elected leader, sending a full tablet report...
I20240502 20:45:25.740998 31624 raft_consensus.cc:374] T aec87a906eaf4295a11cdb346fa7421e P 7f786b85fad644d1be83908499a0c175 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:25.741056 31624 raft_consensus.cc:729] T aec87a906eaf4295a11cdb346fa7421e P 7f786b85fad644d1be83908499a0c175 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f786b85fad644d1be83908499a0c175, State: Initialized, Role: FOLLOWER
I20240502 20:45:25.741051 31623 raft_consensus.cc:729] T aec87a906eaf4295a11cdb346fa7421e P 49d17dbb3e5647f5998cdac93f6011ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49d17dbb3e5647f5998cdac93f6011ae, State: Initialized, Role: FOLLOWER
I20240502 20:45:25.741473 31624 consensus_queue.cc:260] T aec87a906eaf4295a11cdb346fa7421e P 7f786b85fad644d1be83908499a0c175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f786b85fad644d1be83908499a0c175" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 34651 } } peers { permanent_uuid: "49d17dbb3e5647f5998cdac93f6011ae" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 39209 } } peers { permanent_uuid: "77022bea76324055a55a1d7134cf4809" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 34315 } }
I20240502 20:45:25.741473 31623 consensus_queue.cc:260] T aec87a906eaf4295a11cdb346fa7421e P 49d17dbb3e5647f5998cdac93f6011ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f786b85fad644d1be83908499a0c175" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 34651 } } peers { permanent_uuid: "49d17dbb3e5647f5998cdac93f6011ae" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 39209 } } peers { permanent_uuid: "77022bea76324055a55a1d7134cf4809" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 34315 } }
I20240502 20:45:25.742264 31624 ts_tablet_manager.cc:1398] T aec87a906eaf4295a11cdb346fa7421e P 7f786b85fad644d1be83908499a0c175: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240502 20:45:25.742272 31623 ts_tablet_manager.cc:1398] T aec87a906eaf4295a11cdb346fa7421e P 49d17dbb3e5647f5998cdac93f6011ae: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240502 20:45:25.742605 31342 heartbeater.cc:503] Master 127.30.32.126:34843 was elected leader, sending a full tablet report...
I20240502 20:45:25.742625 31080 heartbeater.cc:503] Master 127.30.32.126:34843 was elected leader, sending a full tablet report...
W20240502 20:45:25.824651 31343 tablet.cc:2369] T aec87a906eaf4295a11cdb346fa7421e P 7f786b85fad644d1be83908499a0c175: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 20:45:25.900504 31212 tablet.cc:2369] T aec87a906eaf4295a11cdb346fa7421e P 77022bea76324055a55a1d7134cf4809: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 20:45:25.921258 31629 raft_consensus.cc:482] T aec87a906eaf4295a11cdb346fa7421e P 7f786b85fad644d1be83908499a0c175 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:25.921506 31629 raft_consensus.cc:504] T aec87a906eaf4295a11cdb346fa7421e P 7f786b85fad644d1be83908499a0c175 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f786b85fad644d1be83908499a0c175" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 34651 } } peers { permanent_uuid: "49d17dbb3e5647f5998cdac93f6011ae" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 39209 } } peers { permanent_uuid: "77022bea76324055a55a1d7134cf4809" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 34315 } }
I20240502 20:45:25.924006 31629 leader_election.cc:288] T aec87a906eaf4295a11cdb346fa7421e P 7f786b85fad644d1be83908499a0c175 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49d17dbb3e5647f5998cdac93f6011ae (127.30.32.65:39209), 77022bea76324055a55a1d7134cf4809 (127.30.32.66:34315)
I20240502 20:45:25.928776 31165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aec87a906eaf4295a11cdb346fa7421e" candidate_uuid: "7f786b85fad644d1be83908499a0c175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77022bea76324055a55a1d7134cf4809" is_pre_election: true
I20240502 20:45:25.929180 31165 raft_consensus.cc:2454] T aec87a906eaf4295a11cdb346fa7421e P 77022bea76324055a55a1d7134cf4809 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f786b85fad644d1be83908499a0c175 in term 0.
I20240502 20:45:25.929287 31034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aec87a906eaf4295a11cdb346fa7421e" candidate_uuid: "7f786b85fad644d1be83908499a0c175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49d17dbb3e5647f5998cdac93f6011ae" is_pre_election: true
I20240502 20:45:25.929551 31034 raft_consensus.cc:2454] T aec87a906eaf4295a11cdb346fa7421e P 49d17dbb3e5647f5998cdac93f6011ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f786b85fad644d1be83908499a0c175 in term 0.
I20240502 20:45:25.930094 31230 leader_election.cc:302] T aec87a906eaf4295a11cdb346fa7421e P 7f786b85fad644d1be83908499a0c175 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49d17dbb3e5647f5998cdac93f6011ae, 7f786b85fad644d1be83908499a0c175; no voters:
I20240502 20:45:25.930384 31629 raft_consensus.cc:2781] T aec87a906eaf4295a11cdb346fa7421e P 7f786b85fad644d1be83908499a0c175 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:25.930472 31629 raft_consensus.cc:482] T aec87a906eaf4295a11cdb346fa7421e P 7f786b85fad644d1be83908499a0c175 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:25.930516 31629 raft_consensus.cc:3037] T aec87a906eaf4295a11cdb346fa7421e P 7f786b85fad644d1be83908499a0c175 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:25.931738 31629 raft_consensus.cc:504] T aec87a906eaf4295a11cdb346fa7421e P 7f786b85fad644d1be83908499a0c175 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f786b85fad644d1be83908499a0c175" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 34651 } } peers { permanent_uuid: "49d17dbb3e5647f5998cdac93f6011ae" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 39209 } } peers { permanent_uuid: "77022bea76324055a55a1d7134cf4809" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 34315 } }
I20240502 20:45:25.932145 31629 leader_election.cc:288] T aec87a906eaf4295a11cdb346fa7421e P 7f786b85fad644d1be83908499a0c175 [CANDIDATE]: Term 1 election: Requested vote from peers 49d17dbb3e5647f5998cdac93f6011ae (127.30.32.65:39209), 77022bea76324055a55a1d7134cf4809 (127.30.32.66:34315)
I20240502 20:45:25.932516 31034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aec87a906eaf4295a11cdb346fa7421e" candidate_uuid: "7f786b85fad644d1be83908499a0c175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49d17dbb3e5647f5998cdac93f6011ae"
I20240502 20:45:25.932552 31165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aec87a906eaf4295a11cdb346fa7421e" candidate_uuid: "7f786b85fad644d1be83908499a0c175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77022bea76324055a55a1d7134cf4809"
I20240502 20:45:25.932664 31034 raft_consensus.cc:3037] T aec87a906eaf4295a11cdb346fa7421e P 49d17dbb3e5647f5998cdac93f6011ae [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:25.932699 31165 raft_consensus.cc:3037] T aec87a906eaf4295a11cdb346fa7421e P 77022bea76324055a55a1d7134cf4809 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:25.933799 31034 raft_consensus.cc:2454] T aec87a906eaf4295a11cdb346fa7421e P 49d17dbb3e5647f5998cdac93f6011ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f786b85fad644d1be83908499a0c175 in term 1.
I20240502 20:45:25.934167 31230 leader_election.cc:302] T aec87a906eaf4295a11cdb346fa7421e P 7f786b85fad644d1be83908499a0c175 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49d17dbb3e5647f5998cdac93f6011ae, 7f786b85fad644d1be83908499a0c175; no voters:
I20240502 20:45:25.934376 31629 raft_consensus.cc:2781] T aec87a906eaf4295a11cdb346fa7421e P 7f786b85fad644d1be83908499a0c175 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:25.934629 31629 raft_consensus.cc:686] T aec87a906eaf4295a11cdb346fa7421e P 7f786b85fad644d1be83908499a0c175 [term 1 LEADER]: Becoming Leader. State: Replica: 7f786b85fad644d1be83908499a0c175, State: Running, Role: LEADER
I20240502 20:45:25.935009 31629 consensus_queue.cc:237] T aec87a906eaf4295a11cdb346fa7421e P 7f786b85fad644d1be83908499a0c175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f786b85fad644d1be83908499a0c175" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 34651 } } peers { permanent_uuid: "49d17dbb3e5647f5998cdac93f6011ae" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 39209 } } peers { permanent_uuid: "77022bea76324055a55a1d7134cf4809" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 34315 } }
I20240502 20:45:25.939414 31165 raft_consensus.cc:2454] T aec87a906eaf4295a11cdb346fa7421e P 77022bea76324055a55a1d7134cf4809 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f786b85fad644d1be83908499a0c175 in term 1.
I20240502 20:45:25.940107 30891 catalog_manager.cc:5484] T aec87a906eaf4295a11cdb346fa7421e P 7f786b85fad644d1be83908499a0c175 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f786b85fad644d1be83908499a0c175 (127.30.32.67). New cstate: current_term: 1 leader_uuid: "7f786b85fad644d1be83908499a0c175" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f786b85fad644d1be83908499a0c175" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 34651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49d17dbb3e5647f5998cdac93f6011ae" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 39209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77022bea76324055a55a1d7134cf4809" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 34315 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:25.958299 30849 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 20:45:25.960018 30849 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 49d17dbb3e5647f5998cdac93f6011ae to finish bootstrapping
I20240502 20:45:25.965468 30849 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 77022bea76324055a55a1d7134cf4809 to finish bootstrapping
I20240502 20:45:25.969964 30849 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 7f786b85fad644d1be83908499a0c175 to finish bootstrapping
I20240502 20:45:25.974238 30849 ts_itest-base.cc:233] Waiting for 0 tablets on tserver eba1ba06d3dd42279caed807795fb8ed to finish bootstrapping
W20240502 20:45:25.977542 31081 tablet.cc:2369] T aec87a906eaf4295a11cdb346fa7421e P 49d17dbb3e5647f5998cdac93f6011ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 20:45:25.980288 30849 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 0e3a7d1630044df7ae01d3f144fa4fe3 to finish bootstrapping
I20240502 20:45:26.167831 31145 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:26.168367 31536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:26.169181 31014 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:26.170258 31276 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:45:26.184715 31407 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:26.344303 31629 consensus_queue.cc:1035] T aec87a906eaf4295a11cdb346fa7421e P 7f786b85fad644d1be83908499a0c175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49d17dbb3e5647f5998cdac93f6011ae" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 39209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:26.352320 31629 consensus_queue.cc:1035] T aec87a906eaf4295a11cdb346fa7421e P 7f786b85fad644d1be83908499a0c175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77022bea76324055a55a1d7134cf4809" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 34315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:26.397783 31536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:26.398438 31014 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:26.399430 31276 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:45:26.407975 31145 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:26.408628 31407 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:26.505573 31473 heartbeater.cc:503] Master 127.30.32.126:34843 was elected leader, sending a full tablet report...
I20240502 20:45:26.631646 31014 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:26.632117 31536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:26.635270 31145 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:26.638849 31407 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:26.653515 31276 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:45:26.688639 31605 heartbeater.cc:503] Master 127.30.32.126:34843 was elected leader, sending a full tablet report...
I20240502 20:45:26.731755 31014 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:26.732625 31145 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:26.732636 31536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:26.734186 31407 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:26.745215 31276 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:45:26.839005 31276 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:45:26.838992 31536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:26.841045 31014 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:26.845443 31407 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:26.860177 31145 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:27.016065 30891 ts_manager.cc:292] Set tserver state for 49d17dbb3e5647f5998cdac93f6011ae to MAINTENANCE_MODE
I20240502 20:45:27.208407 31145 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:27.208463 31014 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:27.208676 31276 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:45:27.209298 31407 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:27.213965 31536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:27.294536 31145 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:27.295822 31536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:27.296061 31014 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:27.301522 31276 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:45:27.317529 31407 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:27.427546 31536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:27.428570 31014 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:27.428656 31276 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:45:27.430274 31407 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:27.434048 31145 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:27.466301 31296 consensus_queue.cc:237] T aec87a906eaf4295a11cdb346fa7421e P 7f786b85fad644d1be83908499a0c175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7f786b85fad644d1be83908499a0c175" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 34651 } } peers { permanent_uuid: "49d17dbb3e5647f5998cdac93f6011ae" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 39209 } attrs { replace: true } } peers { permanent_uuid: "77022bea76324055a55a1d7134cf4809" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 34315 } }
I20240502 20:45:27.467648 31165 raft_consensus.cc:1261] T aec87a906eaf4295a11cdb346fa7421e P 77022bea76324055a55a1d7134cf4809 [term 1 FOLLOWER]: Refusing update from remote peer 7f786b85fad644d1be83908499a0c175: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:27.467669 31034 raft_consensus.cc:1261] T aec87a906eaf4295a11cdb346fa7421e P 49d17dbb3e5647f5998cdac93f6011ae [term 1 FOLLOWER]: Refusing update from remote peer 7f786b85fad644d1be83908499a0c175: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:27.468037 31924 consensus_queue.cc:1035] T aec87a906eaf4295a11cdb346fa7421e P 7f786b85fad644d1be83908499a0c175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77022bea76324055a55a1d7134cf4809" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 34315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:45:27.468261 31897 consensus_queue.cc:1035] T aec87a906eaf4295a11cdb346fa7421e P 7f786b85fad644d1be83908499a0c175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49d17dbb3e5647f5998cdac93f6011ae" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 39209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:45:27.470351 31927 raft_consensus.cc:2932] T aec87a906eaf4295a11cdb346fa7421e P 7f786b85fad644d1be83908499a0c175 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7f786b85fad644d1be83908499a0c175" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 34651 } } peers { permanent_uuid: "49d17dbb3e5647f5998cdac93f6011ae" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 39209 } attrs { replace: true } } peers { permanent_uuid: "77022bea76324055a55a1d7134cf4809" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 34315 } } }
I20240502 20:45:27.470603 31034 raft_consensus.cc:2932] T aec87a906eaf4295a11cdb346fa7421e P 49d17dbb3e5647f5998cdac93f6011ae [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7f786b85fad644d1be83908499a0c175" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 34651 } } peers { permanent_uuid: "49d17dbb3e5647f5998cdac93f6011ae" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 39209 } attrs { replace: true } } peers { permanent_uuid: "77022bea76324055a55a1d7134cf4809" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 34315 } } }
I20240502 20:45:27.470656 31165 raft_consensus.cc:2932] T aec87a906eaf4295a11cdb346fa7421e P 77022bea76324055a55a1d7134cf4809 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7f786b85fad644d1be83908499a0c175" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 34651 } } peers { permanent_uuid: "49d17dbb3e5647f5998cdac93f6011ae" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 39209 } attrs { replace: true } } peers { permanent_uuid: "77022bea76324055a55a1d7134cf4809" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 34315 } } }
I20240502 20:45:27.473469 30891 catalog_manager.cc:5484] T aec87a906eaf4295a11cdb346fa7421e P 7f786b85fad644d1be83908499a0c175 reported cstate change: config changed from index -1 to 2. New cstate: current_term: 1 leader_uuid: "7f786b85fad644d1be83908499a0c175" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7f786b85fad644d1be83908499a0c175" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 34651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49d17dbb3e5647f5998cdac93f6011ae" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 39209 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77022bea76324055a55a1d7134cf4809" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 34315 } health_report { overall_health: HEALTHY } } }
I20240502 20:45:27.477090 31296 consensus_queue.cc:237] T aec87a906eaf4295a11cdb346fa7421e P 7f786b85fad644d1be83908499a0c175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7f786b85fad644d1be83908499a0c175" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 34651 } } peers { permanent_uuid: "49d17dbb3e5647f5998cdac93f6011ae" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 39209 } attrs { replace: true } } peers { permanent_uuid: "77022bea76324055a55a1d7134cf4809" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 34315 } } peers { permanent_uuid: "eba1ba06d3dd42279caed807795fb8ed" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 35973 } attrs { promote: true } }
I20240502 20:45:27.478618 31165 raft_consensus.cc:1261] T aec87a906eaf4295a11cdb346fa7421e P 77022bea76324055a55a1d7134cf4809 [term 1 FOLLOWER]: Refusing update from remote peer 7f786b85fad644d1be83908499a0c175: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:45:27.478807 31034 raft_consensus.cc:1261] T aec87a906eaf4295a11cdb346fa7421e P 49d17dbb3e5647f5998cdac93f6011ae [term 1 FOLLOWER]: Refusing update from remote peer 7f786b85fad644d1be83908499a0c175: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:45:27.479060 31927 consensus_queue.cc:1035] T aec87a906eaf4295a11cdb346fa7421e P 7f786b85fad644d1be83908499a0c175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77022bea76324055a55a1d7134cf4809" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 34315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:45:27.479290 31897 consensus_queue.cc:1035] T aec87a906eaf4295a11cdb346fa7421e P 7f786b85fad644d1be83908499a0c175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49d17dbb3e5647f5998cdac93f6011ae" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 39209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:45:27.481489 31897 raft_consensus.cc:2932] T aec87a906eaf4295a11cdb346fa7421e P 7f786b85fad644d1be83908499a0c175 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER eba1ba06d3dd42279caed807795fb8ed (127.30.32.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7f786b85fad644d1be83908499a0c175" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 34651 } } peers { permanent_uuid: "49d17dbb3e5647f5998cdac93f6011ae" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 39209 } attrs { replace: true } } peers { permanent_uuid: "77022bea76324055a55a1d7134cf4809" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 34315 } } peers { permanent_uuid: "eba1ba06d3dd42279caed807795fb8ed" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 35973 } attrs { promote: true } } }
I20240502 20:45:27.482355 31165 raft_consensus.cc:2932] T aec87a906eaf4295a11cdb346fa7421e P 77022bea76324055a55a1d7134cf4809 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER eba1ba06d3dd42279caed807795fb8ed (127.30.32.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7f786b85fad644d1be83908499a0c175" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 34651 } } peers { permanent_uuid: "49d17dbb3e5647f5998cdac93f6011ae" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 39209 } attrs { replace: true } } peers { permanent_uuid: "77022bea76324055a55a1d7134cf4809" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 34315 } } peers { permanent_uuid: "eba1ba06d3dd42279caed807795fb8ed" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 35973 } attrs { promote: true } } }
I20240502 20:45:27.483193 30878 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet aec87a906eaf4295a11cdb346fa7421e with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 20:45:27.483631 31034 raft_consensus.cc:2932] T aec87a906eaf4295a11cdb346fa7421e P 49d17dbb3e5647f5998cdac93f6011ae [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER eba1ba06d3dd42279caed807795fb8ed (127.30.32.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7f786b85fad644d1be83908499a0c175" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 34651 } } peers { permanent_uuid: "49d17dbb3e5647f5998cdac93f6011ae" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 39209 } attrs { replace: true } } peers { permanent_uuid: "77022bea76324055a55a1d7134cf4809" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 34315 } } peers { permanent_uuid: "eba1ba06d3dd42279caed807795fb8ed" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 35973 } attrs { promote: true } } }
I20240502 20:45:27.484462 30891 catalog_manager.cc:5484] T aec87a906eaf4295a11cdb346fa7421e P 7f786b85fad644d1be83908499a0c175 reported cstate change: config changed from index 2 to 3, NON_VOTER eba1ba06d3dd42279caed807795fb8ed (127.30.32.68) added. New cstate: current_term: 1 leader_uuid: "7f786b85fad644d1be83908499a0c175" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7f786b85fad644d1be83908499a0c175" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 34651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49d17dbb3e5647f5998cdac93f6011ae" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 39209 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77022bea76324055a55a1d7134cf4809" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 34315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eba1ba06d3dd42279caed807795fb8ed" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 35973 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 20:45:27.486177 31230 consensus_peers.cc:485] T aec87a906eaf4295a11cdb346fa7421e P 7f786b85fad644d1be83908499a0c175 -> Peer eba1ba06d3dd42279caed807795fb8ed (127.30.32.68:35973): Couldn't send request to peer eba1ba06d3dd42279caed807795fb8ed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aec87a906eaf4295a11cdb346fa7421e. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:28.101601 31951 ts_tablet_manager.cc:902] T aec87a906eaf4295a11cdb346fa7421e P eba1ba06d3dd42279caed807795fb8ed: Initiating tablet copy from peer 7f786b85fad644d1be83908499a0c175 (127.30.32.67:34651)
I20240502 20:45:28.102272 31951 tablet_copy_client.cc:308] T aec87a906eaf4295a11cdb346fa7421e P eba1ba06d3dd42279caed807795fb8ed: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.67:34651
I20240502 20:45:28.105530 31316 tablet_copy_service.cc:140] P 7f786b85fad644d1be83908499a0c175: Received BeginTabletCopySession request for tablet aec87a906eaf4295a11cdb346fa7421e from peer eba1ba06d3dd42279caed807795fb8ed ({username='slave'} at 127.30.32.68:53311)
I20240502 20:45:28.105665 31316 tablet_copy_service.cc:161] P 7f786b85fad644d1be83908499a0c175: Beginning new tablet copy session on tablet aec87a906eaf4295a11cdb346fa7421e from peer eba1ba06d3dd42279caed807795fb8ed at {username='slave'} at 127.30.32.68:53311: session id = eba1ba06d3dd42279caed807795fb8ed-aec87a906eaf4295a11cdb346fa7421e
I20240502 20:45:28.106824 31316 tablet_copy_source_session.cc:205] T aec87a906eaf4295a11cdb346fa7421e P 7f786b85fad644d1be83908499a0c175: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:45:28.109695 31951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aec87a906eaf4295a11cdb346fa7421e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:28.114363 31951 tablet_copy_client.cc:783] T aec87a906eaf4295a11cdb346fa7421e P eba1ba06d3dd42279caed807795fb8ed: tablet copy: Starting download of 0 data blocks...
I20240502 20:45:28.114581 31951 tablet_copy_client.cc:647] T aec87a906eaf4295a11cdb346fa7421e P eba1ba06d3dd42279caed807795fb8ed: tablet copy: Starting download of 1 WAL segments...
I20240502 20:45:28.116176 31951 tablet_copy_client.cc:523] T aec87a906eaf4295a11cdb346fa7421e P eba1ba06d3dd42279caed807795fb8ed: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:45:28.117677 31951 tablet_bootstrap.cc:492] T aec87a906eaf4295a11cdb346fa7421e P eba1ba06d3dd42279caed807795fb8ed: Bootstrap starting.
I20240502 20:45:28.121749 31951 log.cc:824] T aec87a906eaf4295a11cdb346fa7421e P eba1ba06d3dd42279caed807795fb8ed: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:28.124140 31951 tablet_bootstrap.cc:492] T aec87a906eaf4295a11cdb346fa7421e P eba1ba06d3dd42279caed807795fb8ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:45:28.124398 31951 tablet_bootstrap.cc:492] T aec87a906eaf4295a11cdb346fa7421e P eba1ba06d3dd42279caed807795fb8ed: Bootstrap complete.
I20240502 20:45:28.124650 31951 ts_tablet_manager.cc:1367] T aec87a906eaf4295a11cdb346fa7421e P eba1ba06d3dd42279caed807795fb8ed: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240502 20:45:28.127453 31951 raft_consensus.cc:348] T aec87a906eaf4295a11cdb346fa7421e P eba1ba06d3dd42279caed807795fb8ed [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7f786b85fad644d1be83908499a0c175" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 34651 } } peers { permanent_uuid: "49d17dbb3e5647f5998cdac93f6011ae" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 39209 } attrs { replace: true } } peers { permanent_uuid: "77022bea76324055a55a1d7134cf4809" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 34315 } } peers { permanent_uuid: "eba1ba06d3dd42279caed807795fb8ed" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 35973 } attrs { promote: true } }
I20240502 20:45:28.127656 31951 raft_consensus.cc:729] T aec87a906eaf4295a11cdb346fa7421e P eba1ba06d3dd42279caed807795fb8ed [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eba1ba06d3dd42279caed807795fb8ed, State: Initialized, Role: LEARNER
I20240502 20:45:28.127933 31951 consensus_queue.cc:260] T aec87a906eaf4295a11cdb346fa7421e P eba1ba06d3dd42279caed807795fb8ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7f786b85fad644d1be83908499a0c175" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 34651 } } peers { permanent_uuid: "49d17dbb3e5647f5998cdac93f6011ae" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 39209 } attrs { replace: true } } peers { permanent_uuid: "77022bea76324055a55a1d7134cf4809" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 34315 } } peers { permanent_uuid: "eba1ba06d3dd42279caed807795fb8ed" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 35973 } attrs { promote: true } }
I20240502 20:45:28.128715 31951 ts_tablet_manager.cc:1398] T aec87a906eaf4295a11cdb346fa7421e P eba1ba06d3dd42279caed807795fb8ed: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20240502 20:45:28.129432 31316 tablet_copy_service.cc:342] P 7f786b85fad644d1be83908499a0c175: Request end of tablet copy session eba1ba06d3dd42279caed807795fb8ed-aec87a906eaf4295a11cdb346fa7421e received from {username='slave'} at 127.30.32.68:53311
I20240502 20:45:28.129518 31316 tablet_copy_service.cc:433] P 7f786b85fad644d1be83908499a0c175: ending tablet copy session eba1ba06d3dd42279caed807795fb8ed-aec87a906eaf4295a11cdb346fa7421e on tablet aec87a906eaf4295a11cdb346fa7421e with peer eba1ba06d3dd42279caed807795fb8ed
I20240502 20:45:28.574131 31427 raft_consensus.cc:1203] T aec87a906eaf4295a11cdb346fa7421e P eba1ba06d3dd42279caed807795fb8ed [term 1 LEARNER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[]
I20240502 20:45:29.175587 31970 raft_consensus.cc:1050] T aec87a906eaf4295a11cdb346fa7421e P 7f786b85fad644d1be83908499a0c175: attempting to promote NON_VOTER eba1ba06d3dd42279caed807795fb8ed to VOTER
I20240502 20:45:29.176029 31970 consensus_queue.cc:237] T aec87a906eaf4295a11cdb346fa7421e P 7f786b85fad644d1be83908499a0c175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f786b85fad644d1be83908499a0c175" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 34651 } } peers { permanent_uuid: "49d17dbb3e5647f5998cdac93f6011ae" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 39209 } attrs { replace: true } } peers { permanent_uuid: "77022bea76324055a55a1d7134cf4809" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 34315 } } peers { permanent_uuid: "eba1ba06d3dd42279caed807795fb8ed" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 35973 } attrs { promote: false } }
I20240502 20:45:29.177426 31427 raft_consensus.cc:1261] T aec87a906eaf4295a11cdb346fa7421e P eba1ba06d3dd42279caed807795fb8ed [term 1 LEARNER]: Refusing update from remote peer 7f786b85fad644d1be83908499a0c175: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:45:29.177484 31034 raft_consensus.cc:1261] T aec87a906eaf4295a11cdb346fa7421e P 49d17dbb3e5647f5998cdac93f6011ae [term 1 FOLLOWER]: Refusing update from remote peer 7f786b85fad644d1be83908499a0c175: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:45:29.177551 31165 raft_consensus.cc:1261] T aec87a906eaf4295a11cdb346fa7421e P 77022bea76324055a55a1d7134cf4809 [term 1 FOLLOWER]: Refusing update from remote peer 7f786b85fad644d1be83908499a0c175: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:45:29.177872 31986 consensus_queue.cc:1035] T aec87a906eaf4295a11cdb346fa7421e P 7f786b85fad644d1be83908499a0c175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eba1ba06d3dd42279caed807795fb8ed" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 35973 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:45:29.178085 31989 consensus_queue.cc:1035] T aec87a906eaf4295a11cdb346fa7421e P 7f786b85fad644d1be83908499a0c175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49d17dbb3e5647f5998cdac93f6011ae" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 39209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:45:29.178292 31970 consensus_queue.cc:1035] T aec87a906eaf4295a11cdb346fa7421e P 7f786b85fad644d1be83908499a0c175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77022bea76324055a55a1d7134cf4809" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 34315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:45:29.180785 31989 raft_consensus.cc:2932] T aec87a906eaf4295a11cdb346fa7421e P 7f786b85fad644d1be83908499a0c175 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, eba1ba06d3dd42279caed807795fb8ed (127.30.32.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f786b85fad644d1be83908499a0c175" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 34651 } } peers { permanent_uuid: "49d17dbb3e5647f5998cdac93f6011ae" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 39209 } attrs { replace: true } } peers { permanent_uuid: "77022bea76324055a55a1d7134cf4809" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 34315 } } peers { permanent_uuid: "eba1ba06d3dd42279caed807795fb8ed" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 35973 } attrs { promote: false } } }
I20240502 20:45:29.181341 31165 raft_consensus.cc:2932] T aec87a906eaf4295a11cdb346fa7421e P 77022bea76324055a55a1d7134cf4809 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, eba1ba06d3dd42279caed807795fb8ed (127.30.32.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f786b85fad644d1be83908499a0c175" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 34651 } } peers { permanent_uuid: "49d17dbb3e5647f5998cdac93f6011ae" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 39209 } attrs { replace: true } } peers { permanent_uuid: "77022bea76324055a55a1d7134cf4809" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 34315 } } peers { permanent_uuid: "eba1ba06d3dd42279caed807795fb8ed" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 35973 } attrs { promote: false } } }
I20240502 20:45:29.181453 31427 raft_consensus.cc:2932] T aec87a906eaf4295a11cdb346fa7421e P eba1ba06d3dd42279caed807795fb8ed [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, eba1ba06d3dd42279caed807795fb8ed (127.30.32.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f786b85fad644d1be83908499a0c175" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 34651 } } peers { permanent_uuid: "49d17dbb3e5647f5998cdac93f6011ae" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 39209 } attrs { replace: true } } peers { permanent_uuid: "77022bea76324055a55a1d7134cf4809" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 34315 } } peers { permanent_uuid: "eba1ba06d3dd42279caed807795fb8ed" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 35973 } attrs { promote: false } } }
I20240502 20:45:29.183209 31034 raft_consensus.cc:2932] T aec87a906eaf4295a11cdb346fa7421e P 49d17dbb3e5647f5998cdac93f6011ae [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, eba1ba06d3dd42279caed807795fb8ed (127.30.32.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f786b85fad644d1be83908499a0c175" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 34651 } } peers { permanent_uuid: "49d17dbb3e5647f5998cdac93f6011ae" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 39209 } attrs { replace: true } } peers { permanent_uuid: "77022bea76324055a55a1d7134cf4809" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 34315 } } peers { permanent_uuid: "eba1ba06d3dd42279caed807795fb8ed" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 35973 } attrs { promote: false } } }
I20240502 20:45:29.183758 30890 catalog_manager.cc:5484] T aec87a906eaf4295a11cdb346fa7421e P 77022bea76324055a55a1d7134cf4809 reported cstate change: config changed from index 3 to 4, eba1ba06d3dd42279caed807795fb8ed (127.30.32.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7f786b85fad644d1be83908499a0c175" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f786b85fad644d1be83908499a0c175" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 34651 } } peers { permanent_uuid: "49d17dbb3e5647f5998cdac93f6011ae" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 39209 } attrs { replace: true } } peers { permanent_uuid: "77022bea76324055a55a1d7134cf4809" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 34315 } } peers { permanent_uuid: "eba1ba06d3dd42279caed807795fb8ed" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 35973 } attrs { promote: false } } }
I20240502 20:45:29.187453 31296 consensus_queue.cc:237] T aec87a906eaf4295a11cdb346fa7421e P 7f786b85fad644d1be83908499a0c175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f786b85fad644d1be83908499a0c175" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 34651 } } peers { permanent_uuid: "77022bea76324055a55a1d7134cf4809" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 34315 } } peers { permanent_uuid: "eba1ba06d3dd42279caed807795fb8ed" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 35973 } attrs { promote: false } }
I20240502 20:45:29.188509 31165 raft_consensus.cc:1261] T aec87a906eaf4295a11cdb346fa7421e P 77022bea76324055a55a1d7134cf4809 [term 1 FOLLOWER]: Refusing update from remote peer 7f786b85fad644d1be83908499a0c175: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:45:29.188632 31427 raft_consensus.cc:1261] T aec87a906eaf4295a11cdb346fa7421e P eba1ba06d3dd42279caed807795fb8ed [term 1 FOLLOWER]: Refusing update from remote peer 7f786b85fad644d1be83908499a0c175: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:45:29.189013 31970 consensus_queue.cc:1035] T aec87a906eaf4295a11cdb346fa7421e P 7f786b85fad644d1be83908499a0c175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eba1ba06d3dd42279caed807795fb8ed" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 35973 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:45:29.189173 31986 consensus_queue.cc:1035] T aec87a906eaf4295a11cdb346fa7421e P 7f786b85fad644d1be83908499a0c175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77022bea76324055a55a1d7134cf4809" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 34315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:45:29.190622 31989 raft_consensus.cc:2932] T aec87a906eaf4295a11cdb346fa7421e P 7f786b85fad644d1be83908499a0c175 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 49d17dbb3e5647f5998cdac93f6011ae (127.30.32.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f786b85fad644d1be83908499a0c175" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 34651 } } peers { permanent_uuid: "77022bea76324055a55a1d7134cf4809" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 34315 } } peers { permanent_uuid: "eba1ba06d3dd42279caed807795fb8ed" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 35973 } attrs { promote: false } } }
I20240502 20:45:29.191046 31427 raft_consensus.cc:2932] T aec87a906eaf4295a11cdb346fa7421e P eba1ba06d3dd42279caed807795fb8ed [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 49d17dbb3e5647f5998cdac93f6011ae (127.30.32.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f786b85fad644d1be83908499a0c175" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 34651 } } peers { permanent_uuid: "77022bea76324055a55a1d7134cf4809" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 34315 } } peers { permanent_uuid: "eba1ba06d3dd42279caed807795fb8ed" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 35973 } attrs { promote: false } } }
I20240502 20:45:29.191095 31165 raft_consensus.cc:2932] T aec87a906eaf4295a11cdb346fa7421e P 77022bea76324055a55a1d7134cf4809 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 49d17dbb3e5647f5998cdac93f6011ae (127.30.32.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f786b85fad644d1be83908499a0c175" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 34651 } } peers { permanent_uuid: "77022bea76324055a55a1d7134cf4809" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 34315 } } peers { permanent_uuid: "eba1ba06d3dd42279caed807795fb8ed" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 35973 } attrs { promote: false } } }
I20240502 20:45:29.192116 30878 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet aec87a906eaf4295a11cdb346fa7421e with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:45:29.193378 30891 catalog_manager.cc:5484] T aec87a906eaf4295a11cdb346fa7421e P 7f786b85fad644d1be83908499a0c175 reported cstate change: config changed from index 4 to 5, VOTER 49d17dbb3e5647f5998cdac93f6011ae (127.30.32.65) evicted. New cstate: current_term: 1 leader_uuid: "7f786b85fad644d1be83908499a0c175" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f786b85fad644d1be83908499a0c175" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 34651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77022bea76324055a55a1d7134cf4809" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 34315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eba1ba06d3dd42279caed807795fb8ed" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 35973 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:29.195605 31014 tablet_service.cc:1514] Processing DeleteTablet for tablet aec87a906eaf4295a11cdb346fa7421e with delete_type TABLET_DATA_TOMBSTONED (TS 49d17dbb3e5647f5998cdac93f6011ae not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:49628
I20240502 20:45:29.196136 32000 tablet_replica.cc:330] T aec87a906eaf4295a11cdb346fa7421e P 49d17dbb3e5647f5998cdac93f6011ae: stopping tablet replica
I20240502 20:45:29.196413 32000 raft_consensus.cc:2229] T aec87a906eaf4295a11cdb346fa7421e P 49d17dbb3e5647f5998cdac93f6011ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:29.196673 32000 raft_consensus.cc:2258] T aec87a906eaf4295a11cdb346fa7421e P 49d17dbb3e5647f5998cdac93f6011ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:29.197713 32000 ts_tablet_manager.cc:1875] T aec87a906eaf4295a11cdb346fa7421e P 49d17dbb3e5647f5998cdac93f6011ae: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:45:29.199640 32000 ts_tablet_manager.cc:1888] T aec87a906eaf4295a11cdb346fa7421e P 49d17dbb3e5647f5998cdac93f6011ae: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 20:45:29.199724 32000 log.cc:1196] T aec87a906eaf4295a11cdb346fa7421e P 49d17dbb3e5647f5998cdac93f6011ae: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal/wals/aec87a906eaf4295a11cdb346fa7421e
I20240502 20:45:29.200345 30877 catalog_manager.cc:4830] TS 49d17dbb3e5647f5998cdac93f6011ae (127.30.32.65:39209): tablet aec87a906eaf4295a11cdb346fa7421e (table TestTable [id=e8edd2bfffae40a680bfa0a07d381acc]) successfully deleted
I20240502 20:45:29.586577 31014 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:29.586784 31536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:29.587740 31145 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:29.594209 31276 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:45:29.599361 31407 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:29.679322 31014 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:29.681634 31536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:29.683907 31407 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:29.686517 31276 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:45:29.701453 31145 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:29.770047 31536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:29.771138 31014 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:29.773327 31145 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:29.777087 31276 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:45:29.789355 31407 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:29.816500 30849 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskolF7xA/build/debug/bin/kudu with pid 31083
I20240502 20:45:30.002820 31536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:30.003175 31014 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:30.003548 31276 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:45:30.011811 31407 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:30.073829 31407 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:30.073832 31276 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:45:30.073904 31536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:30.074852 31014 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:30.161017 31536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:30.162374 31014 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:30.162897 31276 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:45:30.163442 31407 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:45:30.238916 31231 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.32.66:34315: connect: Connection refused (error 111)
I20240502 20:45:30.239813 31536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:30.240267 31407 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:45:30.240864 31231 consensus_peers.cc:485] T aec87a906eaf4295a11cdb346fa7421e P 7f786b85fad644d1be83908499a0c175 -> Peer 77022bea76324055a55a1d7134cf4809 (127.30.32.66:34315): Couldn't send request to peer 77022bea76324055a55a1d7134cf4809. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.66:34315: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:30.241108 31014 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:30.248090 31276 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:45:30.275558 30849 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskolF7xA/build/debug/bin/kudu with pid 30952
I20240502 20:45:30.281898 30849 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskolF7xA/build/debug/bin/kudu with pid 31214
I20240502 20:45:30.288404 30849 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskolF7xA/build/debug/bin/kudu with pid 31345
I20240502 20:45:30.293967 30849 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskolF7xA/build/debug/bin/kudu with pid 31476
I20240502 20:45:30.298945 30849 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskolF7xA/build/debug/bin/kudu with pid 30861
2024-05-02T20:45:30Z chronyd exiting
[ OK ] RebalanceIgnoredTserversTest.Basic (5759 ms)
[----------] 1 test from RebalanceIgnoredTserversTest (5759 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.TabletServerIsDown/1
I20240502 20:45:30.321403 30849 test_util.cc:268] Using random seed: -1358227977
I20240502 20:45:30.322120 30849 ts_itest-base.cc:112] Starting cluster with:
I20240502 20:45:30.322162 30849 ts_itest-base.cc:113] --------------
I20240502 20:45:30.322181 30849 ts_itest-base.cc:114] 5 tablet servers
I20240502 20:45:30.322196 30849 ts_itest-base.cc:115] 3 replicas per TS
I20240502 20:45:30.322212 30849 ts_itest-base.cc:116] --------------
2024-05-02T20:45:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T20:45:30Z Disabled control of system clock
I20240502 20:45:30.329695 30849 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskolF7xA/build/debug/bin/kudu
/tmp/dist-test-taskolF7xA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.32.126:37465
--webserver_interface=127.30.32.126
--webserver_port=0
--builtin_ntp_servers=127.30.32.84:40883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.32.126:37465
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240502 20:45:30.433012 32222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:30.433251 32222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:30.433315 32222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:30.436235 32222 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:30.436312 32222 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 20:45:30.436347 32222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:30.436386 32222 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 20:45:30.436412 32222 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 20:45:30.440012 32222 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.32.84:40883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.30.32.126:37465
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.32.126:37465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.30.32.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.32222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:30.440918 32222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:30.442061 32222 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:30.449199 32230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:30.449819 32227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:30.450371 32228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:30.451040 32222 server_base.cc:1034] running on GCE node
I20240502 20:45:30.451570 32222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:30.452481 32222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:30.453648 32222 hybrid_clock.cc:648] HybridClock initialized: now 1714682730453617 us; error 44 us; skew 500 ppm
I20240502 20:45:30.455574 32222 webserver.cc:460] Webserver started at http://127.30.32.126:42039/ using document root <none> and password file <none>
I20240502 20:45:30.456128 32222 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:30.456188 32222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:30.456385 32222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:30.458089 32222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "afa89e38968d4529b7405a93971053c3"
format_stamp: "Formatted at 2024-05-02 20:45:30 on dist-test-slave-6336"
I20240502 20:45:30.458559 32222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "afa89e38968d4529b7405a93971053c3"
format_stamp: "Formatted at 2024-05-02 20:45:30 on dist-test-slave-6336"
I20240502 20:45:30.461886 32222 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 20:45:30.463968 32236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:30.465025 32222 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 20:45:30.465142 32222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/wal
uuid: "afa89e38968d4529b7405a93971053c3"
format_stamp: "Formatted at 2024-05-02 20:45:30 on dist-test-slave-6336"
I20240502 20:45:30.465226 32222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:30.476732 32222 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:30.477334 32222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:30.477483 32222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:30.483547 32222 rpc_server.cc:307] RPC server started. Bound to: 127.30.32.126:37465
I20240502 20:45:30.483552 32288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.32.126:37465 every 8 connection(s)
I20240502 20:45:30.484467 32222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 20:45:30.486888 32289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:30.491622 32289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afa89e38968d4529b7405a93971053c3: Bootstrap starting.
I20240502 20:45:30.493769 32289 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P afa89e38968d4529b7405a93971053c3: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:30.494405 32289 log.cc:824] T 00000000000000000000000000000000 P afa89e38968d4529b7405a93971053c3: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:30.494663 30849 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskolF7xA/build/debug/bin/kudu as pid 32222
I20240502 20:45:30.494799 30849 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 20:45:30.496338 32289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afa89e38968d4529b7405a93971053c3: No bootstrap required, opened a new log
I20240502 20:45:30.499302 32289 raft_consensus.cc:348] T 00000000000000000000000000000000 P afa89e38968d4529b7405a93971053c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afa89e38968d4529b7405a93971053c3" member_type: VOTER last_known_addr { host: "127.30.32.126" port: 37465 } }
I20240502 20:45:30.499511 32289 raft_consensus.cc:374] T 00000000000000000000000000000000 P afa89e38968d4529b7405a93971053c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:30.499569 32289 raft_consensus.cc:729] T 00000000000000000000000000000000 P afa89e38968d4529b7405a93971053c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afa89e38968d4529b7405a93971053c3, State: Initialized, Role: FOLLOWER
I20240502 20:45:30.500020 32289 consensus_queue.cc:260] T 00000000000000000000000000000000 P afa89e38968d4529b7405a93971053c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afa89e38968d4529b7405a93971053c3" member_type: VOTER last_known_addr { host: "127.30.32.126" port: 37465 } }
I20240502 20:45:30.500241 32289 raft_consensus.cc:388] T 00000000000000000000000000000000 P afa89e38968d4529b7405a93971053c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 20:45:30.500309 32289 raft_consensus.cc:482] T 00000000000000000000000000000000 P afa89e38968d4529b7405a93971053c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 20:45:30.500408 32289 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afa89e38968d4529b7405a93971053c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:30.501242 32289 raft_consensus.cc:504] T 00000000000000000000000000000000 P afa89e38968d4529b7405a93971053c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afa89e38968d4529b7405a93971053c3" member_type: VOTER last_known_addr { host: "127.30.32.126" port: 37465 } }
I20240502 20:45:30.501587 32289 leader_election.cc:302] T 00000000000000000000000000000000 P afa89e38968d4529b7405a93971053c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: afa89e38968d4529b7405a93971053c3; no voters:
I20240502 20:45:30.501849 32289 leader_election.cc:288] T 00000000000000000000000000000000 P afa89e38968d4529b7405a93971053c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 20:45:30.501935 32294 raft_consensus.cc:2781] T 00000000000000000000000000000000 P afa89e38968d4529b7405a93971053c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:30.502121 32294 raft_consensus.cc:686] T 00000000000000000000000000000000 P afa89e38968d4529b7405a93971053c3 [term 1 LEADER]: Becoming Leader. State: Replica: afa89e38968d4529b7405a93971053c3, State: Running, Role: LEADER
I20240502 20:45:30.502447 32294 consensus_queue.cc:237] T 00000000000000000000000000000000 P afa89e38968d4529b7405a93971053c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afa89e38968d4529b7405a93971053c3" member_type: VOTER last_known_addr { host: "127.30.32.126" port: 37465 } }
I20240502 20:45:30.503064 32289 sys_catalog.cc:564] T 00000000000000000000000000000000 P afa89e38968d4529b7405a93971053c3 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 20:45:30.504565 32295 sys_catalog.cc:455] T 00000000000000000000000000000000 P afa89e38968d4529b7405a93971053c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader afa89e38968d4529b7405a93971053c3. Latest consensus state: current_term: 1 leader_uuid: "afa89e38968d4529b7405a93971053c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afa89e38968d4529b7405a93971053c3" member_type: VOTER last_known_addr { host: "127.30.32.126" port: 37465 } } }
I20240502 20:45:30.504621 32296 sys_catalog.cc:455] T 00000000000000000000000000000000 P afa89e38968d4529b7405a93971053c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "afa89e38968d4529b7405a93971053c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afa89e38968d4529b7405a93971053c3" member_type: VOTER last_known_addr { host: "127.30.32.126" port: 37465 } } }
I20240502 20:45:30.504698 32295 sys_catalog.cc:458] T 00000000000000000000000000000000 P afa89e38968d4529b7405a93971053c3 [sys.catalog]: This master's current role is: LEADER
I20240502 20:45:30.504700 32296 sys_catalog.cc:458] T 00000000000000000000000000000000 P afa89e38968d4529b7405a93971053c3 [sys.catalog]: This master's current role is: LEADER
I20240502 20:45:30.505100 32306 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 20:45:30.507838 32306 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 20:45:30.513715 32306 catalog_manager.cc:1347] Generated new cluster ID: aacc5750b2714ef8a8415b51a9fb31bd
I20240502 20:45:30.513788 32306 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 20:45:30.535635 32306 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 20:45:30.536942 32306 catalog_manager.cc:1504] Loading token signing keys...
I20240502 20:45:30.549911 32306 catalog_manager.cc:5857] T 00000000000000000000000000000000 P afa89e38968d4529b7405a93971053c3: Generated new TSK 0
I20240502 20:45:30.550630 32306 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 20:45:30.553560 30849 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskolF7xA/build/debug/bin/kudu
/tmp/dist-test-taskolF7xA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.32.65:0
--local_ip_for_outbound_sockets=127.30.32.65
--webserver_interface=127.30.32.65
--webserver_port=0
--tserver_master_addrs=127.30.32.126:37465
--builtin_ntp_servers=127.30.32.84:40883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 20:45:30.669754 32313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:30.669975 32313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:30.670042 32313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:30.672971 32313 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:30.673066 32313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:30.673162 32313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.32.65
I20240502 20:45:30.676730 32313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.32.84:40883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.32.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.30.32.65
--webserver_port=0
--tserver_master_addrs=127.30.32.126:37465
--never_fsync=true
--heap_profile_path=/tmp/kudu.32313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.32.65
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:30.677608 32313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:30.678737 32313 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:30.687027 32321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:30.687181 32319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:30.687786 32318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:30.688323 32313 server_base.cc:1034] running on GCE node
I20240502 20:45:30.688799 32313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:30.689565 32313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:30.690732 32313 hybrid_clock.cc:648] HybridClock initialized: now 1714682730690722 us; error 36 us; skew 500 ppm
I20240502 20:45:30.692543 32313 webserver.cc:460] Webserver started at http://127.30.32.65:35913/ using document root <none> and password file <none>
I20240502 20:45:30.693126 32313 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:30.693190 32313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:30.693378 32313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:30.695086 32313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "1d62e5bdea704d548a65cc83999f23d0"
format_stamp: "Formatted at 2024-05-02 20:45:30 on dist-test-slave-6336"
I20240502 20:45:30.695672 32313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "1d62e5bdea704d548a65cc83999f23d0"
format_stamp: "Formatted at 2024-05-02 20:45:30 on dist-test-slave-6336"
I20240502 20:45:30.698949 32313 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 20:45:30.701156 32327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:30.702196 32313 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:30.702316 32313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "1d62e5bdea704d548a65cc83999f23d0"
format_stamp: "Formatted at 2024-05-02 20:45:30 on dist-test-slave-6336"
I20240502 20:45:30.702404 32313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:30.711414 32313 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:30.711938 32313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:30.712088 32313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:30.712566 32313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:30.713485 32313 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:30.713533 32313 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:30.713594 32313 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:30.713620 32313 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:30.721767 32313 rpc_server.cc:307] RPC server started. Bound to: 127.30.32.65:37653
I20240502 20:45:30.721792 32440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.32.65:37653 every 8 connection(s)
I20240502 20:45:30.722723 32313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 20:45:30.729624 30849 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskolF7xA/build/debug/bin/kudu as pid 32313
I20240502 20:45:30.729737 30849 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 20:45:30.731371 32441 heartbeater.cc:347] Connected to a master server at 127.30.32.126:37465
I20240502 20:45:30.731590 32441 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:30.731865 30849 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskolF7xA/build/debug/bin/kudu
/tmp/dist-test-taskolF7xA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.32.66:0
--local_ip_for_outbound_sockets=127.30.32.66
--webserver_interface=127.30.32.66
--webserver_port=0
--tserver_master_addrs=127.30.32.126:37465
--builtin_ntp_servers=127.30.32.84:40883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 20:45:30.732134 32441 heartbeater.cc:511] Master 127.30.32.126:37465 requested a full tablet report, sending...
I20240502 20:45:30.733754 32253 ts_manager.cc:194] Registered new tserver with Master: 1d62e5bdea704d548a65cc83999f23d0 (127.30.32.65:37653)
I20240502 20:45:30.735428 32253 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.32.65:35917
W20240502 20:45:30.840785 32444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:30.841054 32444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:30.841101 32444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:30.844133 32444 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:30.844200 32444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:30.844266 32444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.32.66
I20240502 20:45:30.847870 32444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.32.84:40883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.32.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.30.32.66
--webserver_port=0
--tserver_master_addrs=127.30.32.126:37465
--never_fsync=true
--heap_profile_path=/tmp/kudu.32444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.32.66
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:30.848793 32444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:30.849915 32444 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:30.858498 32450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:30.859072 32452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:30.859265 32449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:30.859941 32444 server_base.cc:1034] running on GCE node
I20240502 20:45:30.860330 32444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:30.860901 32444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:30.862051 32444 hybrid_clock.cc:648] HybridClock initialized: now 1714682730862029 us; error 37 us; skew 500 ppm
I20240502 20:45:30.863943 32444 webserver.cc:460] Webserver started at http://127.30.32.66:46365/ using document root <none> and password file <none>
I20240502 20:45:30.864475 32444 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:30.864532 32444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:30.864715 32444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:30.866400 32444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "51040479a1af45ac962b6954bca33818"
format_stamp: "Formatted at 2024-05-02 20:45:30 on dist-test-slave-6336"
I20240502 20:45:30.866859 32444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "51040479a1af45ac962b6954bca33818"
format_stamp: "Formatted at 2024-05-02 20:45:30 on dist-test-slave-6336"
I20240502 20:45:30.870278 32444 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20240502 20:45:30.872498 32458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:30.873551 32444 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:30.873667 32444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "51040479a1af45ac962b6954bca33818"
format_stamp: "Formatted at 2024-05-02 20:45:30 on dist-test-slave-6336"
I20240502 20:45:30.873765 32444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:30.896365 32444 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:30.897034 32444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:30.897193 32444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:30.897747 32444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:30.898639 32444 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:30.898700 32444 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:30.898759 32444 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:30.898794 32444 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:30.907783 32444 rpc_server.cc:307] RPC server started. Bound to: 127.30.32.66:39443
I20240502 20:45:30.907810 32571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.32.66:39443 every 8 connection(s)
I20240502 20:45:30.908715 32444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 20:45:30.917246 32572 heartbeater.cc:347] Connected to a master server at 127.30.32.126:37465
I20240502 20:45:30.917464 32572 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:30.918002 32572 heartbeater.cc:511] Master 127.30.32.126:37465 requested a full tablet report, sending...
I20240502 20:45:30.918177 30849 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskolF7xA/build/debug/bin/kudu as pid 32444
I20240502 20:45:30.918313 30849 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 20:45:30.919114 32253 ts_manager.cc:194] Registered new tserver with Master: 51040479a1af45ac962b6954bca33818 (127.30.32.66:39443)
I20240502 20:45:30.919677 32253 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.32.66:41085
I20240502 20:45:30.920480 30849 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskolF7xA/build/debug/bin/kudu
/tmp/dist-test-taskolF7xA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.32.67:0
--local_ip_for_outbound_sockets=127.30.32.67
--webserver_interface=127.30.32.67
--webserver_port=0
--tserver_master_addrs=127.30.32.126:37465
--builtin_ntp_servers=127.30.32.84:40883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 20:45:31.027642 32575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:31.027871 32575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:31.027938 32575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:31.030925 32575 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:31.030995 32575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:31.031080 32575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.32.67
I20240502 20:45:31.034610 32575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.32.84:40883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.32.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.30.32.67
--webserver_port=0
--tserver_master_addrs=127.30.32.126:37465
--never_fsync=true
--heap_profile_path=/tmp/kudu.32575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.32.67
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:31.035465 32575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:31.036533 32575 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:31.044237 32583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:31.044891 32580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:31.045396 32581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:31.045953 32575 server_base.cc:1034] running on GCE node
I20240502 20:45:31.046309 32575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:31.046826 32575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:31.047987 32575 hybrid_clock.cc:648] HybridClock initialized: now 1714682731047972 us; error 30 us; skew 500 ppm
I20240502 20:45:31.049944 32575 webserver.cc:460] Webserver started at http://127.30.32.67:41639/ using document root <none> and password file <none>
I20240502 20:45:31.050477 32575 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:31.050536 32575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:31.050700 32575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:31.052410 32575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "cc1c9afcd56249ffabc5dea1c20336ac"
format_stamp: "Formatted at 2024-05-02 20:45:31 on dist-test-slave-6336"
I20240502 20:45:31.052924 32575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "cc1c9afcd56249ffabc5dea1c20336ac"
format_stamp: "Formatted at 2024-05-02 20:45:31 on dist-test-slave-6336"
I20240502 20:45:31.056245 32575 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240502 20:45:31.058413 32589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:31.059368 32575 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 20:45:31.059474 32575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "cc1c9afcd56249ffabc5dea1c20336ac"
format_stamp: "Formatted at 2024-05-02 20:45:31 on dist-test-slave-6336"
I20240502 20:45:31.059557 32575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:31.079456 32575 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:31.080085 32575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:31.080248 32575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:31.080785 32575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:31.081728 32575 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:31.081782 32575 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:31.081843 32575 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:31.081874 32575 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:31.090045 32575 rpc_server.cc:307] RPC server started. Bound to: 127.30.32.67:37551
I20240502 20:45:31.090065 32702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.32.67:37551 every 8 connection(s)
I20240502 20:45:31.090965 32575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 20:45:31.096012 30849 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskolF7xA/build/debug/bin/kudu as pid 32575
I20240502 20:45:31.096099 30849 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 20:45:31.098090 30849 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskolF7xA/build/debug/bin/kudu
/tmp/dist-test-taskolF7xA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.32.68:0
--local_ip_for_outbound_sockets=127.30.32.68
--webserver_interface=127.30.32.68
--webserver_port=0
--tserver_master_addrs=127.30.32.126:37465
--builtin_ntp_servers=127.30.32.84:40883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 20:45:31.099789 32703 heartbeater.cc:347] Connected to a master server at 127.30.32.126:37465
I20240502 20:45:31.100003 32703 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:31.100510 32703 heartbeater.cc:511] Master 127.30.32.126:37465 requested a full tablet report, sending...
I20240502 20:45:31.101539 32253 ts_manager.cc:194] Registered new tserver with Master: cc1c9afcd56249ffabc5dea1c20336ac (127.30.32.67:37551)
I20240502 20:45:31.102067 32253 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.32.67:43989
W20240502 20:45:31.207010 32706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:31.207264 32706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:31.207346 32706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:31.211650 32706 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:31.211737 32706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:31.211835 32706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.32.68
I20240502 20:45:31.215586 32706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.32.84:40883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.32.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.30.32.68
--webserver_port=0
--tserver_master_addrs=127.30.32.126:37465
--never_fsync=true
--heap_profile_path=/tmp/kudu.32706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.32.68
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:31.216536 32706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:31.217705 32706 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:31.225622 32714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:31.226315 32711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:31.226898 32712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:31.227490 32706 server_base.cc:1034] running on GCE node
I20240502 20:45:31.227845 32706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:31.228349 32706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:31.229491 32706 hybrid_clock.cc:648] HybridClock initialized: now 1714682731229470 us; error 35 us; skew 500 ppm
I20240502 20:45:31.231376 32706 webserver.cc:460] Webserver started at http://127.30.32.68:33327/ using document root <none> and password file <none>
I20240502 20:45:31.231909 32706 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:31.231967 32706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:31.232157 32706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:31.233822 32706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "bcfa3791e74046d18b911c73e9c5abf3"
format_stamp: "Formatted at 2024-05-02 20:45:31 on dist-test-slave-6336"
I20240502 20:45:31.234285 32706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "bcfa3791e74046d18b911c73e9c5abf3"
format_stamp: "Formatted at 2024-05-02 20:45:31 on dist-test-slave-6336"
I20240502 20:45:31.237435 32706 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 20:45:31.239444 32720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:31.240363 32706 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240502 20:45:31.240458 32706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "bcfa3791e74046d18b911c73e9c5abf3"
format_stamp: "Formatted at 2024-05-02 20:45:31 on dist-test-slave-6336"
I20240502 20:45:31.240541 32706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:31.255746 32706 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:31.256284 32706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:31.256433 32706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:31.256937 32706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:31.257812 32706 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:31.257859 32706 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:31.257918 32706 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:31.257947 32706 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:31.265991 32706 rpc_server.cc:307] RPC server started. Bound to: 127.30.32.68:41499
I20240502 20:45:31.266010 365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.32.68:41499 every 8 connection(s)
I20240502 20:45:31.266875 32706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 20:45:31.274134 30849 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskolF7xA/build/debug/bin/kudu as pid 32706
I20240502 20:45:31.274255 30849 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 20:45:31.275225 366 heartbeater.cc:347] Connected to a master server at 127.30.32.126:37465
I20240502 20:45:31.275440 366 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:31.275969 366 heartbeater.cc:511] Master 127.30.32.126:37465 requested a full tablet report, sending...
I20240502 20:45:31.276197 30849 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskolF7xA/build/debug/bin/kudu
/tmp/dist-test-taskolF7xA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.32.69:0
--local_ip_for_outbound_sockets=127.30.32.69
--webserver_interface=127.30.32.69
--webserver_port=0
--tserver_master_addrs=127.30.32.126:37465
--builtin_ntp_servers=127.30.32.84:40883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 20:45:31.276855 32253 ts_manager.cc:194] Registered new tserver with Master: bcfa3791e74046d18b911c73e9c5abf3 (127.30.32.68:41499)
I20240502 20:45:31.277499 32253 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.32.68:37835
W20240502 20:45:31.379349 369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:31.379571 369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:31.379619 369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:31.382781 369 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:31.382843 369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:31.382930 369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.32.69
I20240502 20:45:31.386561 369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.32.84:40883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.32.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.30.32.69
--webserver_port=0
--tserver_master_addrs=127.30.32.126:37465
--never_fsync=true
--heap_profile_path=/tmp/kudu.369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.32.69
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:31.387424 369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:31.388454 369 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:31.396364 374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:31.396965 377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:31.397504 375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:31.398113 369 server_base.cc:1034] running on GCE node
I20240502 20:45:31.398471 369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:31.398983 369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:31.400128 369 hybrid_clock.cc:648] HybridClock initialized: now 1714682731400115 us; error 28 us; skew 500 ppm
I20240502 20:45:31.401947 369 webserver.cc:460] Webserver started at http://127.30.32.69:40301/ using document root <none> and password file <none>
I20240502 20:45:31.402489 369 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:31.402544 369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:31.402725 369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:31.404348 369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "a4c87a45e47c4558a4ea9e52246f9a66"
format_stamp: "Formatted at 2024-05-02 20:45:31 on dist-test-slave-6336"
I20240502 20:45:31.404821 369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "a4c87a45e47c4558a4ea9e52246f9a66"
format_stamp: "Formatted at 2024-05-02 20:45:31 on dist-test-slave-6336"
I20240502 20:45:31.408080 369 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20240502 20:45:31.410205 383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:31.411156 369 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 20:45:31.411252 369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "a4c87a45e47c4558a4ea9e52246f9a66"
format_stamp: "Formatted at 2024-05-02 20:45:31 on dist-test-slave-6336"
I20240502 20:45:31.411330 369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:31.427809 369 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:31.428431 369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:31.428587 369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:31.429159 369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:31.430042 369 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:31.430090 369 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:31.430150 369 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:31.430176 369 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:31.438360 369 rpc_server.cc:307] RPC server started. Bound to: 127.30.32.69:40285
I20240502 20:45:31.438385 496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.32.69:40285 every 8 connection(s)
I20240502 20:45:31.439285 369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 20:45:31.441648 30849 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskolF7xA/build/debug/bin/kudu as pid 369
I20240502 20:45:31.441767 30849 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 20:45:31.447981 497 heartbeater.cc:347] Connected to a master server at 127.30.32.126:37465
I20240502 20:45:31.448173 497 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:31.448643 497 heartbeater.cc:511] Master 127.30.32.126:37465 requested a full tablet report, sending...
I20240502 20:45:31.449532 32253 ts_manager.cc:194] Registered new tserver with Master: a4c87a45e47c4558a4ea9e52246f9a66 (127.30.32.69:40285)
I20240502 20:45:31.450049 32253 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.32.69:58503
I20240502 20:45:31.455722 30849 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 20:45:31.464295 32253 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57056:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 20:45:31.482750 32637 tablet_service.cc:1467] Processing CreateTablet for tablet 4ba0ce4cde70449b80abcd7504b9774d (DEFAULT_TABLE table=TestTable [id=57d4aaec2d6e45729ace2e584059291d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:31.483104 32375 tablet_service.cc:1467] Processing CreateTablet for tablet 4ba0ce4cde70449b80abcd7504b9774d (DEFAULT_TABLE table=TestTable [id=57d4aaec2d6e45729ace2e584059291d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:31.483700 32637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ba0ce4cde70449b80abcd7504b9774d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:31.483994 32375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ba0ce4cde70449b80abcd7504b9774d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:31.483834 32506 tablet_service.cc:1467] Processing CreateTablet for tablet 4ba0ce4cde70449b80abcd7504b9774d (DEFAULT_TABLE table=TestTable [id=57d4aaec2d6e45729ace2e584059291d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:31.484608 32506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ba0ce4cde70449b80abcd7504b9774d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:31.490561 515 tablet_bootstrap.cc:492] T 4ba0ce4cde70449b80abcd7504b9774d P 1d62e5bdea704d548a65cc83999f23d0: Bootstrap starting.
I20240502 20:45:31.490926 516 tablet_bootstrap.cc:492] T 4ba0ce4cde70449b80abcd7504b9774d P 51040479a1af45ac962b6954bca33818: Bootstrap starting.
I20240502 20:45:31.492632 516 tablet_bootstrap.cc:654] T 4ba0ce4cde70449b80abcd7504b9774d P 51040479a1af45ac962b6954bca33818: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:31.493283 514 tablet_bootstrap.cc:492] T 4ba0ce4cde70449b80abcd7504b9774d P cc1c9afcd56249ffabc5dea1c20336ac: Bootstrap starting.
I20240502 20:45:31.493393 516 log.cc:824] T 4ba0ce4cde70449b80abcd7504b9774d P 51040479a1af45ac962b6954bca33818: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:31.495218 516 tablet_bootstrap.cc:492] T 4ba0ce4cde70449b80abcd7504b9774d P 51040479a1af45ac962b6954bca33818: No bootstrap required, opened a new log
I20240502 20:45:31.495309 515 tablet_bootstrap.cc:654] T 4ba0ce4cde70449b80abcd7504b9774d P 1d62e5bdea704d548a65cc83999f23d0: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:31.495404 516 ts_tablet_manager.cc:1367] T 4ba0ce4cde70449b80abcd7504b9774d P 51040479a1af45ac962b6954bca33818: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 20:45:31.495719 514 tablet_bootstrap.cc:654] T 4ba0ce4cde70449b80abcd7504b9774d P cc1c9afcd56249ffabc5dea1c20336ac: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:31.496042 515 log.cc:824] T 4ba0ce4cde70449b80abcd7504b9774d P 1d62e5bdea704d548a65cc83999f23d0: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:31.496381 514 log.cc:824] T 4ba0ce4cde70449b80abcd7504b9774d P cc1c9afcd56249ffabc5dea1c20336ac: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:31.497910 515 tablet_bootstrap.cc:492] T 4ba0ce4cde70449b80abcd7504b9774d P 1d62e5bdea704d548a65cc83999f23d0: No bootstrap required, opened a new log
I20240502 20:45:31.498064 515 ts_tablet_manager.cc:1367] T 4ba0ce4cde70449b80abcd7504b9774d P 1d62e5bdea704d548a65cc83999f23d0: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.001s
I20240502 20:45:31.498159 514 tablet_bootstrap.cc:492] T 4ba0ce4cde70449b80abcd7504b9774d P cc1c9afcd56249ffabc5dea1c20336ac: No bootstrap required, opened a new log
I20240502 20:45:31.498327 514 ts_tablet_manager.cc:1367] T 4ba0ce4cde70449b80abcd7504b9774d P cc1c9afcd56249ffabc5dea1c20336ac: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 20:45:31.499230 516 raft_consensus.cc:348] T 4ba0ce4cde70449b80abcd7504b9774d P 51040479a1af45ac962b6954bca33818 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d62e5bdea704d548a65cc83999f23d0" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 37653 } } peers { permanent_uuid: "cc1c9afcd56249ffabc5dea1c20336ac" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 37551 } } peers { permanent_uuid: "51040479a1af45ac962b6954bca33818" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 39443 } }
I20240502 20:45:31.499454 516 raft_consensus.cc:374] T 4ba0ce4cde70449b80abcd7504b9774d P 51040479a1af45ac962b6954bca33818 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:31.499500 516 raft_consensus.cc:729] T 4ba0ce4cde70449b80abcd7504b9774d P 51040479a1af45ac962b6954bca33818 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51040479a1af45ac962b6954bca33818, State: Initialized, Role: FOLLOWER
I20240502 20:45:31.500236 516 consensus_queue.cc:260] T 4ba0ce4cde70449b80abcd7504b9774d P 51040479a1af45ac962b6954bca33818 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d62e5bdea704d548a65cc83999f23d0" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 37653 } } peers { permanent_uuid: "cc1c9afcd56249ffabc5dea1c20336ac" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 37551 } } peers { permanent_uuid: "51040479a1af45ac962b6954bca33818" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 39443 } }
I20240502 20:45:31.501222 516 ts_tablet_manager.cc:1398] T 4ba0ce4cde70449b80abcd7504b9774d P 51040479a1af45ac962b6954bca33818: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240502 20:45:31.501564 32572 heartbeater.cc:503] Master 127.30.32.126:37465 was elected leader, sending a full tablet report...
I20240502 20:45:31.501968 514 raft_consensus.cc:348] T 4ba0ce4cde70449b80abcd7504b9774d P cc1c9afcd56249ffabc5dea1c20336ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d62e5bdea704d548a65cc83999f23d0" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 37653 } } peers { permanent_uuid: "cc1c9afcd56249ffabc5dea1c20336ac" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 37551 } } peers { permanent_uuid: "51040479a1af45ac962b6954bca33818" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 39443 } }
I20240502 20:45:31.501945 515 raft_consensus.cc:348] T 4ba0ce4cde70449b80abcd7504b9774d P 1d62e5bdea704d548a65cc83999f23d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d62e5bdea704d548a65cc83999f23d0" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 37653 } } peers { permanent_uuid: "cc1c9afcd56249ffabc5dea1c20336ac" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 37551 } } peers { permanent_uuid: "51040479a1af45ac962b6954bca33818" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 39443 } }
I20240502 20:45:31.502127 514 raft_consensus.cc:374] T 4ba0ce4cde70449b80abcd7504b9774d P cc1c9afcd56249ffabc5dea1c20336ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:31.502136 515 raft_consensus.cc:374] T 4ba0ce4cde70449b80abcd7504b9774d P 1d62e5bdea704d548a65cc83999f23d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:31.502187 515 raft_consensus.cc:729] T 4ba0ce4cde70449b80abcd7504b9774d P 1d62e5bdea704d548a65cc83999f23d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d62e5bdea704d548a65cc83999f23d0, State: Initialized, Role: FOLLOWER
I20240502 20:45:31.502207 514 raft_consensus.cc:729] T 4ba0ce4cde70449b80abcd7504b9774d P cc1c9afcd56249ffabc5dea1c20336ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc1c9afcd56249ffabc5dea1c20336ac, State: Initialized, Role: FOLLOWER
I20240502 20:45:31.502591 514 consensus_queue.cc:260] T 4ba0ce4cde70449b80abcd7504b9774d P cc1c9afcd56249ffabc5dea1c20336ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d62e5bdea704d548a65cc83999f23d0" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 37653 } } peers { permanent_uuid: "cc1c9afcd56249ffabc5dea1c20336ac" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 37551 } } peers { permanent_uuid: "51040479a1af45ac962b6954bca33818" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 39443 } }
I20240502 20:45:31.502641 515 consensus_queue.cc:260] T 4ba0ce4cde70449b80abcd7504b9774d P 1d62e5bdea704d548a65cc83999f23d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d62e5bdea704d548a65cc83999f23d0" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 37653 } } peers { permanent_uuid: "cc1c9afcd56249ffabc5dea1c20336ac" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 37551 } } peers { permanent_uuid: "51040479a1af45ac962b6954bca33818" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 39443 } }
I20240502 20:45:31.503173 32703 heartbeater.cc:503] Master 127.30.32.126:37465 was elected leader, sending a full tablet report...
I20240502 20:45:31.503384 514 ts_tablet_manager.cc:1398] T 4ba0ce4cde70449b80abcd7504b9774d P cc1c9afcd56249ffabc5dea1c20336ac: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240502 20:45:31.503536 515 ts_tablet_manager.cc:1398] T 4ba0ce4cde70449b80abcd7504b9774d P 1d62e5bdea704d548a65cc83999f23d0: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20240502 20:45:31.503875 32441 heartbeater.cc:503] Master 127.30.32.126:37465 was elected leader, sending a full tablet report...
I20240502 20:45:31.553295 520 raft_consensus.cc:482] T 4ba0ce4cde70449b80abcd7504b9774d P 51040479a1af45ac962b6954bca33818 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:31.553512 520 raft_consensus.cc:504] T 4ba0ce4cde70449b80abcd7504b9774d P 51040479a1af45ac962b6954bca33818 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d62e5bdea704d548a65cc83999f23d0" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 37653 } } peers { permanent_uuid: "cc1c9afcd56249ffabc5dea1c20336ac" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 37551 } } peers { permanent_uuid: "51040479a1af45ac962b6954bca33818" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 39443 } }
I20240502 20:45:31.554687 520 leader_election.cc:288] T 4ba0ce4cde70449b80abcd7504b9774d P 51040479a1af45ac962b6954bca33818 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d62e5bdea704d548a65cc83999f23d0 (127.30.32.65:37653), cc1c9afcd56249ffabc5dea1c20336ac (127.30.32.67:37551)
I20240502 20:45:31.559414 32657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba0ce4cde70449b80abcd7504b9774d" candidate_uuid: "51040479a1af45ac962b6954bca33818" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc1c9afcd56249ffabc5dea1c20336ac" is_pre_election: true
I20240502 20:45:31.559417 32395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba0ce4cde70449b80abcd7504b9774d" candidate_uuid: "51040479a1af45ac962b6954bca33818" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d62e5bdea704d548a65cc83999f23d0" is_pre_election: true
I20240502 20:45:31.559757 32395 raft_consensus.cc:2454] T 4ba0ce4cde70449b80abcd7504b9774d P 1d62e5bdea704d548a65cc83999f23d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51040479a1af45ac962b6954bca33818 in term 0.
I20240502 20:45:31.559769 32657 raft_consensus.cc:2454] T 4ba0ce4cde70449b80abcd7504b9774d P cc1c9afcd56249ffabc5dea1c20336ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51040479a1af45ac962b6954bca33818 in term 0.
I20240502 20:45:31.560261 32460 leader_election.cc:302] T 4ba0ce4cde70449b80abcd7504b9774d P 51040479a1af45ac962b6954bca33818 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51040479a1af45ac962b6954bca33818, cc1c9afcd56249ffabc5dea1c20336ac; no voters:
I20240502 20:45:31.560504 520 raft_consensus.cc:2781] T 4ba0ce4cde70449b80abcd7504b9774d P 51040479a1af45ac962b6954bca33818 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:31.560588 520 raft_consensus.cc:482] T 4ba0ce4cde70449b80abcd7504b9774d P 51040479a1af45ac962b6954bca33818 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:31.560663 520 raft_consensus.cc:3037] T 4ba0ce4cde70449b80abcd7504b9774d P 51040479a1af45ac962b6954bca33818 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:31.574322 520 raft_consensus.cc:504] T 4ba0ce4cde70449b80abcd7504b9774d P 51040479a1af45ac962b6954bca33818 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d62e5bdea704d548a65cc83999f23d0" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 37653 } } peers { permanent_uuid: "cc1c9afcd56249ffabc5dea1c20336ac" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 37551 } } peers { permanent_uuid: "51040479a1af45ac962b6954bca33818" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 39443 } }
I20240502 20:45:31.575138 520 leader_election.cc:288] T 4ba0ce4cde70449b80abcd7504b9774d P 51040479a1af45ac962b6954bca33818 [CANDIDATE]: Term 1 election: Requested vote from peers 1d62e5bdea704d548a65cc83999f23d0 (127.30.32.65:37653), cc1c9afcd56249ffabc5dea1c20336ac (127.30.32.67:37551)
I20240502 20:45:31.575655 32395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba0ce4cde70449b80abcd7504b9774d" candidate_uuid: "51040479a1af45ac962b6954bca33818" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d62e5bdea704d548a65cc83999f23d0"
I20240502 20:45:31.575655 32657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba0ce4cde70449b80abcd7504b9774d" candidate_uuid: "51040479a1af45ac962b6954bca33818" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc1c9afcd56249ffabc5dea1c20336ac"
I20240502 20:45:31.575807 32657 raft_consensus.cc:3037] T 4ba0ce4cde70449b80abcd7504b9774d P cc1c9afcd56249ffabc5dea1c20336ac [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:31.575806 32395 raft_consensus.cc:3037] T 4ba0ce4cde70449b80abcd7504b9774d P 1d62e5bdea704d548a65cc83999f23d0 [term 0 FOLLOWER]: Advancing to term 1
W20240502 20:45:31.592041 32704 tablet.cc:2369] T 4ba0ce4cde70449b80abcd7504b9774d P cc1c9afcd56249ffabc5dea1c20336ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 20:45:31.606981 32395 raft_consensus.cc:2454] T 4ba0ce4cde70449b80abcd7504b9774d P 1d62e5bdea704d548a65cc83999f23d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51040479a1af45ac962b6954bca33818 in term 1.
I20240502 20:45:31.606981 32657 raft_consensus.cc:2454] T 4ba0ce4cde70449b80abcd7504b9774d P cc1c9afcd56249ffabc5dea1c20336ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51040479a1af45ac962b6954bca33818 in term 1.
I20240502 20:45:31.607465 32460 leader_election.cc:302] T 4ba0ce4cde70449b80abcd7504b9774d P 51040479a1af45ac962b6954bca33818 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51040479a1af45ac962b6954bca33818, cc1c9afcd56249ffabc5dea1c20336ac; no voters:
I20240502 20:45:31.607687 520 raft_consensus.cc:2781] T 4ba0ce4cde70449b80abcd7504b9774d P 51040479a1af45ac962b6954bca33818 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:31.607956 520 raft_consensus.cc:686] T 4ba0ce4cde70449b80abcd7504b9774d P 51040479a1af45ac962b6954bca33818 [term 1 LEADER]: Becoming Leader. State: Replica: 51040479a1af45ac962b6954bca33818, State: Running, Role: LEADER
I20240502 20:45:31.608319 520 consensus_queue.cc:237] T 4ba0ce4cde70449b80abcd7504b9774d P 51040479a1af45ac962b6954bca33818 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d62e5bdea704d548a65cc83999f23d0" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 37653 } } peers { permanent_uuid: "cc1c9afcd56249ffabc5dea1c20336ac" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 37551 } } peers { permanent_uuid: "51040479a1af45ac962b6954bca33818" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 39443 } }
I20240502 20:45:31.611450 32253 catalog_manager.cc:5484] T 4ba0ce4cde70449b80abcd7504b9774d P 51040479a1af45ac962b6954bca33818 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51040479a1af45ac962b6954bca33818 (127.30.32.66). New cstate: current_term: 1 leader_uuid: "51040479a1af45ac962b6954bca33818" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d62e5bdea704d548a65cc83999f23d0" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 37653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc1c9afcd56249ffabc5dea1c20336ac" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 37551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51040479a1af45ac962b6954bca33818" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 39443 } health_report { overall_health: HEALTHY } } }
I20240502 20:45:31.632900 30849 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 20:45:31.634263 30849 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 1d62e5bdea704d548a65cc83999f23d0 to finish bootstrapping
I20240502 20:45:31.640559 30849 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 51040479a1af45ac962b6954bca33818 to finish bootstrapping
I20240502 20:45:31.645045 30849 ts_itest-base.cc:233] Waiting for 1 tablets on tserver cc1c9afcd56249ffabc5dea1c20336ac to finish bootstrapping
I20240502 20:45:31.649506 30849 ts_itest-base.cc:233] Waiting for 0 tablets on tserver bcfa3791e74046d18b911c73e9c5abf3 to finish bootstrapping
I20240502 20:45:31.655002 30849 ts_itest-base.cc:233] Waiting for 0 tablets on tserver a4c87a45e47c4558a4ea9e52246f9a66 to finish bootstrapping
W20240502 20:45:31.660058 32573 tablet.cc:2369] T 4ba0ce4cde70449b80abcd7504b9774d P 51040479a1af45ac962b6954bca33818: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 20:45:31.660701 30849 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskolF7xA/build/debug/bin/kudu with pid 32313
I20240502 20:45:31.855419 300 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:31.855513 431 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:31.856458 32506 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:45:31.858240 32637 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:31.935616 32506 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:45:31.936306 32637 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:31.936502 300 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:31.937451 431 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:32.034617 32506 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:45:32.034912 431 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:32.036175 32637 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:32.041338 300 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:32.146306 618 consensus_queue.cc:1035] T 4ba0ce4cde70449b80abcd7504b9774d P 51040479a1af45ac962b6954bca33818 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc1c9afcd56249ffabc5dea1c20336ac" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 37551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 20:45:32.151630 32460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.32.65:37653: connect: Connection refused (error 111)
W20240502 20:45:32.152840 32460 consensus_peers.cc:485] T 4ba0ce4cde70449b80abcd7504b9774d P 51040479a1af45ac962b6954bca33818 -> Peer 1d62e5bdea704d548a65cc83999f23d0 (127.30.32.65:37653): Couldn't send request to peer 1d62e5bdea704d548a65cc83999f23d0. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.65:37653: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:32.259704 431 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:32.260322 32506 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:45:32.260766 300 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:32.261044 32637 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:32.279358 366 heartbeater.cc:503] Master 127.30.32.126:37465 was elected leader, sending a full tablet report...
I20240502 20:45:32.336629 32506 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:45:32.337052 431 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:32.337265 32637 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:32.337707 300 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:32.431735 300 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:32.432000 431 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:32.432200 32506 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:45:32.435285 32637 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:32.451858 497 heartbeater.cc:503] Master 127.30.32.126:37465 was elected leader, sending a full tablet report...
I20240502 20:45:32.471885 30849 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskolF7xA/build/debug/bin/kudu with pid 32444
I20240502 20:45:32.477761 30849 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskolF7xA/build/debug/bin/kudu with pid 32575
I20240502 20:45:32.483881 30849 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskolF7xA/build/debug/bin/kudu with pid 32706
I20240502 20:45:32.489459 30849 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskolF7xA/build/debug/bin/kudu with pid 369
I20240502 20:45:32.496268 30849 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskolF7xA/build/debug/bin/kudu with pid 32222
2024-05-02T20:45:32Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.TabletServerIsDown/1 (2197 ms)
[----------] 1 test from RebalanceStartCriteriaTest (2197 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/1
I20240502 20:45:32.519026 30849 test_util.cc:268] Using random seed: -1356030355
I20240502 20:45:32.519767 30849 ts_itest-base.cc:112] Starting cluster with:
I20240502 20:45:32.519810 30849 ts_itest-base.cc:113] --------------
I20240502 20:45:32.519829 30849 ts_itest-base.cc:114] 7 tablet servers
I20240502 20:45:32.519845 30849 ts_itest-base.cc:115] 1 replicas per TS
I20240502 20:45:32.519872 30849 ts_itest-base.cc:116] --------------
2024-05-02T20:45:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T20:45:32Z Disabled control of system clock
I20240502 20:45:32.527882 30849 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskolF7xA/build/debug/bin/kudu
/tmp/dist-test-taskolF7xA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.32.126:43021
--webserver_interface=127.30.32.126
--webserver_port=0
--builtin_ntp_servers=127.30.32.84:35991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.32.126:43021
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240502 20:45:32.640676 720 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240502 20:45:32.641016 720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:32.641072 720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:32.641146 720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:32.644431 720 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:32.644521 720 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 20:45:32.644570 720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:32.644616 720 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 20:45:32.644657 720 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 20:45:32.648444 720 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.32.84:35991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.30.32.126:43021
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.32.126:43021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.30.32.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:32.649405 720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:32.650549 720 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:32.658125 725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:32.658682 728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:32.659322 726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:32.659955 720 server_base.cc:1034] running on GCE node
I20240502 20:45:32.660437 720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:32.661386 720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:32.662532 720 hybrid_clock.cc:648] HybridClock initialized: now 1714682732662519 us; error 26 us; skew 500 ppm
I20240502 20:45:32.664362 720 webserver.cc:460] Webserver started at http://127.30.32.126:40605/ using document root <none> and password file <none>
I20240502 20:45:32.664909 720 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:32.664970 720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:32.665203 720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:32.666828 720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "5c223b22641f4581b6ec8c5e553014cd"
format_stamp: "Formatted at 2024-05-02 20:45:32 on dist-test-slave-6336"
I20240502 20:45:32.667284 720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "5c223b22641f4581b6ec8c5e553014cd"
format_stamp: "Formatted at 2024-05-02 20:45:32 on dist-test-slave-6336"
I20240502 20:45:32.670547 720 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20240502 20:45:32.672964 734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:32.674014 720 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 20:45:32.674141 720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/wal
uuid: "5c223b22641f4581b6ec8c5e553014cd"
format_stamp: "Formatted at 2024-05-02 20:45:32 on dist-test-slave-6336"
I20240502 20:45:32.674239 720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:32.711021 720 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:32.711720 720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:32.711870 720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:32.718690 720 rpc_server.cc:307] RPC server started. Bound to: 127.30.32.126:43021
I20240502 20:45:32.718686 786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.32.126:43021 every 8 connection(s)
I20240502 20:45:32.719724 720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 20:45:32.722460 787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:32.723534 30849 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskolF7xA/build/debug/bin/kudu as pid 720
I20240502 20:45:32.723657 30849 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 20:45:32.728183 787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c223b22641f4581b6ec8c5e553014cd: Bootstrap starting.
I20240502 20:45:32.730377 787 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c223b22641f4581b6ec8c5e553014cd: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:32.731225 787 log.cc:824] T 00000000000000000000000000000000 P 5c223b22641f4581b6ec8c5e553014cd: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:32.732973 787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c223b22641f4581b6ec8c5e553014cd: No bootstrap required, opened a new log
I20240502 20:45:32.735402 787 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c223b22641f4581b6ec8c5e553014cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c223b22641f4581b6ec8c5e553014cd" member_type: VOTER last_known_addr { host: "127.30.32.126" port: 43021 } }
I20240502 20:45:32.735580 787 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c223b22641f4581b6ec8c5e553014cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:32.735638 787 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c223b22641f4581b6ec8c5e553014cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c223b22641f4581b6ec8c5e553014cd, State: Initialized, Role: FOLLOWER
I20240502 20:45:32.736023 787 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c223b22641f4581b6ec8c5e553014cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c223b22641f4581b6ec8c5e553014cd" member_type: VOTER last_known_addr { host: "127.30.32.126" port: 43021 } }
I20240502 20:45:32.736147 787 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c223b22641f4581b6ec8c5e553014cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 20:45:32.736208 787 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c223b22641f4581b6ec8c5e553014cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 20:45:32.736310 787 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c223b22641f4581b6ec8c5e553014cd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:32.782016 787 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c223b22641f4581b6ec8c5e553014cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c223b22641f4581b6ec8c5e553014cd" member_type: VOTER last_known_addr { host: "127.30.32.126" port: 43021 } }
I20240502 20:45:32.782593 787 leader_election.cc:302] T 00000000000000000000000000000000 P 5c223b22641f4581b6ec8c5e553014cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c223b22641f4581b6ec8c5e553014cd; no voters:
I20240502 20:45:32.783005 787 leader_election.cc:288] T 00000000000000000000000000000000 P 5c223b22641f4581b6ec8c5e553014cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 20:45:32.783418 792 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c223b22641f4581b6ec8c5e553014cd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:32.783666 792 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c223b22641f4581b6ec8c5e553014cd [term 1 LEADER]: Becoming Leader. State: Replica: 5c223b22641f4581b6ec8c5e553014cd, State: Running, Role: LEADER
I20240502 20:45:32.784061 792 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c223b22641f4581b6ec8c5e553014cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c223b22641f4581b6ec8c5e553014cd" member_type: VOTER last_known_addr { host: "127.30.32.126" port: 43021 } }
I20240502 20:45:32.784482 787 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c223b22641f4581b6ec8c5e553014cd [sys.catalog]: configured and running, proceeding with master startup.
I20240502 20:45:32.785827 793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c223b22641f4581b6ec8c5e553014cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c223b22641f4581b6ec8c5e553014cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c223b22641f4581b6ec8c5e553014cd" member_type: VOTER last_known_addr { host: "127.30.32.126" port: 43021 } } }
I20240502 20:45:32.785952 793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c223b22641f4581b6ec8c5e553014cd [sys.catalog]: This master's current role is: LEADER
I20240502 20:45:32.786338 794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c223b22641f4581b6ec8c5e553014cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c223b22641f4581b6ec8c5e553014cd. Latest consensus state: current_term: 1 leader_uuid: "5c223b22641f4581b6ec8c5e553014cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c223b22641f4581b6ec8c5e553014cd" member_type: VOTER last_known_addr { host: "127.30.32.126" port: 43021 } } }
I20240502 20:45:32.786496 794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c223b22641f4581b6ec8c5e553014cd [sys.catalog]: This master's current role is: LEADER
I20240502 20:45:32.786823 801 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 20:45:32.789927 801 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 20:45:32.795708 801 catalog_manager.cc:1347] Generated new cluster ID: 565423534b814f64b1dd1e9cd7addc39
I20240502 20:45:32.795814 801 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 20:45:32.801586 801 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 20:45:32.802544 801 catalog_manager.cc:1504] Loading token signing keys...
I20240502 20:45:32.808320 801 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 5c223b22641f4581b6ec8c5e553014cd: Generated new TSK 0
I20240502 20:45:32.809213 801 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 20:45:32.819804 30849 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskolF7xA/build/debug/bin/kudu
/tmp/dist-test-taskolF7xA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.32.65:0
--local_ip_for_outbound_sockets=127.30.32.65
--webserver_interface=127.30.32.65
--webserver_port=0
--tserver_master_addrs=127.30.32.126:43021
--builtin_ntp_servers=127.30.32.84:35991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 20:45:32.932210 811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:32.932426 811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:32.932471 811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:32.935550 811 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:32.935621 811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:32.935711 811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.32.65
I20240502 20:45:32.939338 811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.32.84:35991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.32.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.30.32.65
--webserver_port=0
--tserver_master_addrs=127.30.32.126:43021
--never_fsync=true
--heap_profile_path=/tmp/kudu.811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.32.65
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:32.940248 811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:32.941442 811 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:32.950018 819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:32.950536 817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:32.951185 816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:32.951941 811 server_base.cc:1034] running on GCE node
I20240502 20:45:32.952329 811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:32.952898 811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:32.954046 811 hybrid_clock.cc:648] HybridClock initialized: now 1714682732954026 us; error 34 us; skew 500 ppm
I20240502 20:45:32.955842 811 webserver.cc:460] Webserver started at http://127.30.32.65:40123/ using document root <none> and password file <none>
I20240502 20:45:32.956384 811 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:32.956440 811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:32.956627 811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:32.958360 811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "33be9a5b29214a8d84c2fd7a4936a920"
format_stamp: "Formatted at 2024-05-02 20:45:32 on dist-test-slave-6336"
I20240502 20:45:32.958851 811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "33be9a5b29214a8d84c2fd7a4936a920"
format_stamp: "Formatted at 2024-05-02 20:45:32 on dist-test-slave-6336"
I20240502 20:45:32.962316 811 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 20:45:32.964570 825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:32.965617 811 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:32.965709 811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "33be9a5b29214a8d84c2fd7a4936a920"
format_stamp: "Formatted at 2024-05-02 20:45:32 on dist-test-slave-6336"
I20240502 20:45:32.965807 811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:32.981285 811 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:32.981935 811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:32.982075 811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:32.982584 811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:32.983551 811 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:32.983601 811 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:32.983642 811 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:32.983664 811 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:32.993048 811 rpc_server.cc:307] RPC server started. Bound to: 127.30.32.65:45537
I20240502 20:45:32.993091 938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.32.65:45537 every 8 connection(s)
I20240502 20:45:32.994020 811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 20:45:32.996495 30849 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskolF7xA/build/debug/bin/kudu as pid 811
I20240502 20:45:32.996600 30849 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 20:45:32.998809 30849 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskolF7xA/build/debug/bin/kudu
/tmp/dist-test-taskolF7xA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.32.66:0
--local_ip_for_outbound_sockets=127.30.32.66
--webserver_interface=127.30.32.66
--webserver_port=0
--tserver_master_addrs=127.30.32.126:43021
--builtin_ntp_servers=127.30.32.84:35991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 20:45:33.003759 939 heartbeater.cc:347] Connected to a master server at 127.30.32.126:43021
I20240502 20:45:33.004051 939 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:33.004787 939 heartbeater.cc:511] Master 127.30.32.126:43021 requested a full tablet report, sending...
I20240502 20:45:33.006424 746 ts_manager.cc:194] Registered new tserver with Master: 33be9a5b29214a8d84c2fd7a4936a920 (127.30.32.65:45537)
I20240502 20:45:33.007665 746 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.32.65:50899
W20240502 20:45:33.112090 942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:33.112339 942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:33.112392 942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:33.115550 942 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:33.115636 942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:33.115739 942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.32.66
I20240502 20:45:33.119616 942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.32.84:35991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.32.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.30.32.66
--webserver_port=0
--tserver_master_addrs=127.30.32.126:43021
--never_fsync=true
--heap_profile_path=/tmp/kudu.942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.32.66
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:33.120692 942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:33.122004 942 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:33.130470 947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:33.130964 950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:33.131599 948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:33.132215 942 server_base.cc:1034] running on GCE node
I20240502 20:45:33.132566 942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:33.133172 942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:33.134316 942 hybrid_clock.cc:648] HybridClock initialized: now 1714682733134304 us; error 31 us; skew 500 ppm
I20240502 20:45:33.136029 942 webserver.cc:460] Webserver started at http://127.30.32.66:36771/ using document root <none> and password file <none>
I20240502 20:45:33.136549 942 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:33.136603 942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:33.136792 942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:33.138455 942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "ed5783865eb74d39bee2a38e9139dea4"
format_stamp: "Formatted at 2024-05-02 20:45:33 on dist-test-slave-6336"
I20240502 20:45:33.138926 942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "ed5783865eb74d39bee2a38e9139dea4"
format_stamp: "Formatted at 2024-05-02 20:45:33 on dist-test-slave-6336"
I20240502 20:45:33.142089 942 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 20:45:33.144119 956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:33.145104 942 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 20:45:33.145205 942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "ed5783865eb74d39bee2a38e9139dea4"
format_stamp: "Formatted at 2024-05-02 20:45:33 on dist-test-slave-6336"
I20240502 20:45:33.145287 942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:33.159490 942 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:33.160086 942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:33.160240 942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:33.160742 942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:33.161623 942 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:33.161674 942 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:33.161731 942 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:33.161768 942 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:33.169957 942 rpc_server.cc:307] RPC server started. Bound to: 127.30.32.66:45113
I20240502 20:45:33.169976 1069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.32.66:45113 every 8 connection(s)
I20240502 20:45:33.170871 942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 20:45:33.174597 30849 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskolF7xA/build/debug/bin/kudu as pid 942
I20240502 20:45:33.174703 30849 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 20:45:33.176944 30849 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskolF7xA/build/debug/bin/kudu
/tmp/dist-test-taskolF7xA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.32.67:0
--local_ip_for_outbound_sockets=127.30.32.67
--webserver_interface=127.30.32.67
--webserver_port=0
--tserver_master_addrs=127.30.32.126:43021
--builtin_ntp_servers=127.30.32.84:35991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 20:45:33.179919 1070 heartbeater.cc:347] Connected to a master server at 127.30.32.126:43021
I20240502 20:45:33.180124 1070 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:33.180624 1070 heartbeater.cc:511] Master 127.30.32.126:43021 requested a full tablet report, sending...
I20240502 20:45:33.181682 746 ts_manager.cc:194] Registered new tserver with Master: ed5783865eb74d39bee2a38e9139dea4 (127.30.32.66:45113)
I20240502 20:45:33.182523 746 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.32.66:33605
W20240502 20:45:33.287194 1073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:33.287433 1073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:33.287503 1073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:33.290760 1073 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:33.290844 1073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:33.290915 1073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.32.67
I20240502 20:45:33.294739 1073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.32.84:35991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.32.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.30.32.67
--webserver_port=0
--tserver_master_addrs=127.30.32.126:43021
--never_fsync=true
--heap_profile_path=/tmp/kudu.1073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.32.67
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:33.295711 1073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:33.297461 1073 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:33.306200 1078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:33.306712 1081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:33.307371 1079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:33.308131 1073 server_base.cc:1034] running on GCE node
I20240502 20:45:33.308562 1073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:33.309211 1073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:33.310364 1073 hybrid_clock.cc:648] HybridClock initialized: now 1714682733310353 us; error 24 us; skew 500 ppm
I20240502 20:45:33.312144 1073 webserver.cc:460] Webserver started at http://127.30.32.67:41483/ using document root <none> and password file <none>
I20240502 20:45:33.312736 1073 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:33.312789 1073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:33.312999 1073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:33.314688 1073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "f7fd7bf659f84ae184696b891db5221a"
format_stamp: "Formatted at 2024-05-02 20:45:33 on dist-test-slave-6336"
I20240502 20:45:33.315141 1073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "f7fd7bf659f84ae184696b891db5221a"
format_stamp: "Formatted at 2024-05-02 20:45:33 on dist-test-slave-6336"
I20240502 20:45:33.318536 1073 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 20:45:33.320698 1087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:33.321753 1073 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:33.321862 1073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "f7fd7bf659f84ae184696b891db5221a"
format_stamp: "Formatted at 2024-05-02 20:45:33 on dist-test-slave-6336"
I20240502 20:45:33.321952 1073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:33.344214 1073 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:33.344882 1073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:33.345070 1073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:33.345623 1073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:33.346527 1073 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:33.346575 1073 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:33.346623 1073 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:33.346653 1073 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:33.355221 1073 rpc_server.cc:307] RPC server started. Bound to: 127.30.32.67:42407
I20240502 20:45:33.355248 1200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.32.67:42407 every 8 connection(s)
I20240502 20:45:33.356120 1073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 20:45:33.363541 30849 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskolF7xA/build/debug/bin/kudu as pid 1073
I20240502 20:45:33.363657 30849 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 20:45:33.364444 1201 heartbeater.cc:347] Connected to a master server at 127.30.32.126:43021
I20240502 20:45:33.364678 1201 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:33.365296 1201 heartbeater.cc:511] Master 127.30.32.126:43021 requested a full tablet report, sending...
I20240502 20:45:33.365703 30849 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskolF7xA/build/debug/bin/kudu
/tmp/dist-test-taskolF7xA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.32.68:0
--local_ip_for_outbound_sockets=127.30.32.68
--webserver_interface=127.30.32.68
--webserver_port=0
--tserver_master_addrs=127.30.32.126:43021
--builtin_ntp_servers=127.30.32.84:35991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 20:45:33.366338 746 ts_manager.cc:194] Registered new tserver with Master: f7fd7bf659f84ae184696b891db5221a (127.30.32.67:42407)
I20240502 20:45:33.367085 746 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.32.67:52999
W20240502 20:45:33.476825 1204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:33.477085 1204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:33.477149 1204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:33.480542 1204 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:33.480626 1204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:33.480731 1204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.32.68
I20240502 20:45:33.484413 1204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.32.84:35991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.32.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.30.32.68
--webserver_port=0
--tserver_master_addrs=127.30.32.126:43021
--never_fsync=true
--heap_profile_path=/tmp/kudu.1204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.32.68
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:33.485536 1204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:33.486821 1204 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:33.495554 1209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:33.496079 1210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:33.496742 1212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:33.497601 1204 server_base.cc:1034] running on GCE node
I20240502 20:45:33.497975 1204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:33.498529 1204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:33.499679 1204 hybrid_clock.cc:648] HybridClock initialized: now 1714682733499651 us; error 48 us; skew 500 ppm
I20240502 20:45:33.501542 1204 webserver.cc:460] Webserver started at http://127.30.32.68:42261/ using document root <none> and password file <none>
I20240502 20:45:33.502108 1204 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:33.502171 1204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:33.502350 1204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:33.504132 1204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "2260afde4ccb498189d7b25c00a2c727"
format_stamp: "Formatted at 2024-05-02 20:45:33 on dist-test-slave-6336"
I20240502 20:45:33.504621 1204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "2260afde4ccb498189d7b25c00a2c727"
format_stamp: "Formatted at 2024-05-02 20:45:33 on dist-test-slave-6336"
I20240502 20:45:33.508055 1204 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20240502 20:45:33.510378 1218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:33.511389 1204 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 20:45:33.511504 1204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "2260afde4ccb498189d7b25c00a2c727"
format_stamp: "Formatted at 2024-05-02 20:45:33 on dist-test-slave-6336"
I20240502 20:45:33.511593 1204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:33.538401 1204 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:33.539077 1204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:33.539245 1204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:33.539811 1204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:33.540724 1204 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:33.540786 1204 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:33.540848 1204 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:33.540874 1204 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:33.549774 1204 rpc_server.cc:307] RPC server started. Bound to: 127.30.32.68:40723
I20240502 20:45:33.549803 1331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.32.68:40723 every 8 connection(s)
I20240502 20:45:33.550731 1204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 20:45:33.552202 30849 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskolF7xA/build/debug/bin/kudu as pid 1204
I20240502 20:45:33.552300 30849 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 20:45:33.554471 30849 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskolF7xA/build/debug/bin/kudu
/tmp/dist-test-taskolF7xA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.32.69:0
--local_ip_for_outbound_sockets=127.30.32.69
--webserver_interface=127.30.32.69
--webserver_port=0
--tserver_master_addrs=127.30.32.126:43021
--builtin_ntp_servers=127.30.32.84:35991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 20:45:33.560432 1332 heartbeater.cc:347] Connected to a master server at 127.30.32.126:43021
I20240502 20:45:33.560672 1332 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:33.561239 1332 heartbeater.cc:511] Master 127.30.32.126:43021 requested a full tablet report, sending...
I20240502 20:45:33.562258 746 ts_manager.cc:194] Registered new tserver with Master: 2260afde4ccb498189d7b25c00a2c727 (127.30.32.68:40723)
I20240502 20:45:33.563068 746 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.32.68:48423
W20240502 20:45:33.665295 1335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:33.665511 1335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:33.665577 1335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:33.668578 1335 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:33.668653 1335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:33.668718 1335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.32.69
I20240502 20:45:33.672397 1335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.32.84:35991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.32.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.30.32.69
--webserver_port=0
--tserver_master_addrs=127.30.32.126:43021
--never_fsync=true
--heap_profile_path=/tmp/kudu.1335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.32.69
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:33.673369 1335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:33.674451 1335 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:33.683038 1343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:33.683451 1340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:33.684116 1341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:33.684836 1335 server_base.cc:1034] running on GCE node
I20240502 20:45:33.685232 1335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:33.685907 1335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:33.687134 1335 hybrid_clock.cc:648] HybridClock initialized: now 1714682733687120 us; error 30 us; skew 500 ppm
I20240502 20:45:33.688966 1335 webserver.cc:460] Webserver started at http://127.30.32.69:43257/ using document root <none> and password file <none>
I20240502 20:45:33.689718 1335 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:33.689793 1335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:33.689957 1335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:33.691601 1335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "cec988e98db64e2f98b030efc16d4108"
format_stamp: "Formatted at 2024-05-02 20:45:33 on dist-test-slave-6336"
I20240502 20:45:33.692065 1335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "cec988e98db64e2f98b030efc16d4108"
format_stamp: "Formatted at 2024-05-02 20:45:33 on dist-test-slave-6336"
I20240502 20:45:33.695618 1335 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20240502 20:45:33.698016 1349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:33.699151 1335 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 20:45:33.699285 1335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "cec988e98db64e2f98b030efc16d4108"
format_stamp: "Formatted at 2024-05-02 20:45:33 on dist-test-slave-6336"
I20240502 20:45:33.699394 1335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:33.715234 1335 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:33.715937 1335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:33.716102 1335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:33.716630 1335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:33.717554 1335 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:33.717604 1335 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:33.717667 1335 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:33.717698 1335 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:33.727396 1335 rpc_server.cc:307] RPC server started. Bound to: 127.30.32.69:34789
I20240502 20:45:33.727458 1462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.32.69:34789 every 8 connection(s)
I20240502 20:45:33.728384 1335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 20:45:33.730234 30849 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskolF7xA/build/debug/bin/kudu as pid 1335
I20240502 20:45:33.730337 30849 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 20:45:33.732568 30849 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskolF7xA/build/debug/bin/kudu
/tmp/dist-test-taskolF7xA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.32.70:0
--local_ip_for_outbound_sockets=127.30.32.70
--webserver_interface=127.30.32.70
--webserver_port=0
--tserver_master_addrs=127.30.32.126:43021
--builtin_ntp_servers=127.30.32.84:35991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 20:45:33.738361 1463 heartbeater.cc:347] Connected to a master server at 127.30.32.126:43021
I20240502 20:45:33.738602 1463 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:33.739156 1463 heartbeater.cc:511] Master 127.30.32.126:43021 requested a full tablet report, sending...
I20240502 20:45:33.740165 746 ts_manager.cc:194] Registered new tserver with Master: cec988e98db64e2f98b030efc16d4108 (127.30.32.69:34789)
I20240502 20:45:33.740739 746 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.32.69:33441
W20240502 20:45:33.841451 1466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:33.841696 1466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:33.841768 1466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:33.844743 1466 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:33.844825 1466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:33.844918 1466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.32.70
I20240502 20:45:33.848570 1466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.32.84:35991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.32.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.30.32.70
--webserver_port=0
--tserver_master_addrs=127.30.32.126:43021
--never_fsync=true
--heap_profile_path=/tmp/kudu.1466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.32.70
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:33.849527 1466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:33.850692 1466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:33.859285 1474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:33.859771 1471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:33.860455 1472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:33.861189 1466 server_base.cc:1034] running on GCE node
I20240502 20:45:33.861559 1466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:33.862149 1466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:33.863303 1466 hybrid_clock.cc:648] HybridClock initialized: now 1714682733863292 us; error 23 us; skew 500 ppm
I20240502 20:45:33.865285 1466 webserver.cc:460] Webserver started at http://127.30.32.70:41997/ using document root <none> and password file <none>
I20240502 20:45:33.865897 1466 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:33.865968 1466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:33.866168 1466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:33.868126 1466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "6d231fa7f1684535af46a9b196111880"
format_stamp: "Formatted at 2024-05-02 20:45:33 on dist-test-slave-6336"
I20240502 20:45:33.868664 1466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "6d231fa7f1684535af46a9b196111880"
format_stamp: "Formatted at 2024-05-02 20:45:33 on dist-test-slave-6336"
I20240502 20:45:33.872242 1466 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20240502 20:45:33.874722 1480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:33.875730 1466 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 20:45:33.875849 1466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "6d231fa7f1684535af46a9b196111880"
format_stamp: "Formatted at 2024-05-02 20:45:33 on dist-test-slave-6336"
I20240502 20:45:33.875941 1466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:33.892341 1466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:33.893213 1466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:33.893376 1466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:33.893936 1466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:33.894881 1466 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:33.894934 1466 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:33.894997 1466 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:33.895030 1466 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:33.903678 1466 rpc_server.cc:307] RPC server started. Bound to: 127.30.32.70:43291
I20240502 20:45:33.903717 1593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.32.70:43291 every 8 connection(s)
I20240502 20:45:33.904676 1466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 20:45:33.909020 30849 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskolF7xA/build/debug/bin/kudu as pid 1466
I20240502 20:45:33.909152 30849 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 20:45:33.911592 30849 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskolF7xA/build/debug/bin/kudu
/tmp/dist-test-taskolF7xA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.32.71:0
--local_ip_for_outbound_sockets=127.30.32.71
--webserver_interface=127.30.32.71
--webserver_port=0
--tserver_master_addrs=127.30.32.126:43021
--builtin_ntp_servers=127.30.32.84:35991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 20:45:33.913897 1594 heartbeater.cc:347] Connected to a master server at 127.30.32.126:43021
I20240502 20:45:33.914127 1594 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:33.914778 1594 heartbeater.cc:511] Master 127.30.32.126:43021 requested a full tablet report, sending...
I20240502 20:45:33.915940 746 ts_manager.cc:194] Registered new tserver with Master: 6d231fa7f1684535af46a9b196111880 (127.30.32.70:43291)
I20240502 20:45:33.916534 746 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.32.70:57101
I20240502 20:45:34.010663 939 heartbeater.cc:503] Master 127.30.32.126:43021 was elected leader, sending a full tablet report...
W20240502 20:45:34.020244 1597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:34.020449 1597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:34.020505 1597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:34.023965 1597 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:34.024048 1597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:34.024155 1597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.32.71
I20240502 20:45:34.028249 1597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.32.84:35991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.32.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.30.32.71
--webserver_port=0
--tserver_master_addrs=127.30.32.126:43021
--never_fsync=true
--heap_profile_path=/tmp/kudu.1597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.32.71
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:34.029173 1597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:34.030273 1597 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:34.038638 1602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:34.039177 1605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:34.039871 1603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:34.040628 1597 server_base.cc:1034] running on GCE node
I20240502 20:45:34.041136 1597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:34.041700 1597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:34.042855 1597 hybrid_clock.cc:648] HybridClock initialized: now 1714682734042827 us; error 47 us; skew 500 ppm
I20240502 20:45:34.044757 1597 webserver.cc:460] Webserver started at http://127.30.32.71:42625/ using document root <none> and password file <none>
I20240502 20:45:34.045344 1597 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:34.045408 1597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:34.045583 1597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:34.047260 1597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "0dfc68b63d15462398c76bfbff0e3cf8"
format_stamp: "Formatted at 2024-05-02 20:45:34 on dist-test-slave-6336"
I20240502 20:45:34.047739 1597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "0dfc68b63d15462398c76bfbff0e3cf8"
format_stamp: "Formatted at 2024-05-02 20:45:34 on dist-test-slave-6336"
I20240502 20:45:34.051091 1597 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20240502 20:45:34.053372 1611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:34.054360 1597 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 20:45:34.054467 1597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "0dfc68b63d15462398c76bfbff0e3cf8"
format_stamp: "Formatted at 2024-05-02 20:45:34 on dist-test-slave-6336"
I20240502 20:45:34.054548 1597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:34.102265 1597 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:34.102959 1597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:34.103119 1597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:34.103730 1597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:34.104652 1597 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:34.104702 1597 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:34.104761 1597 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:34.104789 1597 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:34.114794 1597 rpc_server.cc:307] RPC server started. Bound to: 127.30.32.71:40439
I20240502 20:45:34.114832 1724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.32.71:40439 every 8 connection(s)
I20240502 20:45:34.115787 1597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 20:45:34.117892 30849 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskolF7xA/build/debug/bin/kudu as pid 1597
I20240502 20:45:34.118022 30849 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 20:45:34.124819 1725 heartbeater.cc:347] Connected to a master server at 127.30.32.126:43021
I20240502 20:45:34.125083 1725 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:34.125635 1725 heartbeater.cc:511] Master 127.30.32.126:43021 requested a full tablet report, sending...
I20240502 20:45:34.126598 746 ts_manager.cc:194] Registered new tserver with Master: 0dfc68b63d15462398c76bfbff0e3cf8 (127.30.32.71:40439)
I20240502 20:45:34.127125 746 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.32.71:54345
I20240502 20:45:34.132048 30849 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 20:45:34.140775 746 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53992:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 20:45:34.157881 1266 tablet_service.cc:1467] Processing CreateTablet for tablet e7080f04f87d4a6f926edea263b450bc (DEFAULT_TABLE table=TestTable [id=0d544ebaaf6c45ba84e5a294e388a7f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:34.158823 1266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7080f04f87d4a6f926edea263b450bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:34.164234 1739 tablet_bootstrap.cc:492] T e7080f04f87d4a6f926edea263b450bc P 2260afde4ccb498189d7b25c00a2c727: Bootstrap starting.
I20240502 20:45:34.166266 1739 tablet_bootstrap.cc:654] T e7080f04f87d4a6f926edea263b450bc P 2260afde4ccb498189d7b25c00a2c727: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:34.166968 1739 log.cc:824] T e7080f04f87d4a6f926edea263b450bc P 2260afde4ccb498189d7b25c00a2c727: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:34.168768 1739 tablet_bootstrap.cc:492] T e7080f04f87d4a6f926edea263b450bc P 2260afde4ccb498189d7b25c00a2c727: No bootstrap required, opened a new log
I20240502 20:45:34.168944 1739 ts_tablet_manager.cc:1367] T e7080f04f87d4a6f926edea263b450bc P 2260afde4ccb498189d7b25c00a2c727: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 20:45:34.171533 1739 raft_consensus.cc:348] T e7080f04f87d4a6f926edea263b450bc P 2260afde4ccb498189d7b25c00a2c727 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2260afde4ccb498189d7b25c00a2c727" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 40723 } }
I20240502 20:45:34.171711 1739 raft_consensus.cc:374] T e7080f04f87d4a6f926edea263b450bc P 2260afde4ccb498189d7b25c00a2c727 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:34.171753 1739 raft_consensus.cc:729] T e7080f04f87d4a6f926edea263b450bc P 2260afde4ccb498189d7b25c00a2c727 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2260afde4ccb498189d7b25c00a2c727, State: Initialized, Role: FOLLOWER
I20240502 20:45:34.172183 1739 consensus_queue.cc:260] T e7080f04f87d4a6f926edea263b450bc P 2260afde4ccb498189d7b25c00a2c727 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2260afde4ccb498189d7b25c00a2c727" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 40723 } }
I20240502 20:45:34.172354 1739 raft_consensus.cc:388] T e7080f04f87d4a6f926edea263b450bc P 2260afde4ccb498189d7b25c00a2c727 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 20:45:34.172420 1739 raft_consensus.cc:482] T e7080f04f87d4a6f926edea263b450bc P 2260afde4ccb498189d7b25c00a2c727 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 20:45:34.172526 1739 raft_consensus.cc:3037] T e7080f04f87d4a6f926edea263b450bc P 2260afde4ccb498189d7b25c00a2c727 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:34.184479 1070 heartbeater.cc:503] Master 127.30.32.126:43021 was elected leader, sending a full tablet report...
I20240502 20:45:34.211683 1739 raft_consensus.cc:504] T e7080f04f87d4a6f926edea263b450bc P 2260afde4ccb498189d7b25c00a2c727 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2260afde4ccb498189d7b25c00a2c727" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 40723 } }
I20240502 20:45:34.212218 1739 leader_election.cc:302] T e7080f04f87d4a6f926edea263b450bc P 2260afde4ccb498189d7b25c00a2c727 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2260afde4ccb498189d7b25c00a2c727; no voters:
I20240502 20:45:34.212764 1739 leader_election.cc:288] T e7080f04f87d4a6f926edea263b450bc P 2260afde4ccb498189d7b25c00a2c727 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 20:45:34.212900 1741 raft_consensus.cc:2781] T e7080f04f87d4a6f926edea263b450bc P 2260afde4ccb498189d7b25c00a2c727 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:34.213344 1741 raft_consensus.cc:686] T e7080f04f87d4a6f926edea263b450bc P 2260afde4ccb498189d7b25c00a2c727 [term 1 LEADER]: Becoming Leader. State: Replica: 2260afde4ccb498189d7b25c00a2c727, State: Running, Role: LEADER
I20240502 20:45:34.213786 1741 consensus_queue.cc:237] T e7080f04f87d4a6f926edea263b450bc P 2260afde4ccb498189d7b25c00a2c727 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2260afde4ccb498189d7b25c00a2c727" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 40723 } }
I20240502 20:45:34.214017 1739 ts_tablet_manager.cc:1398] T e7080f04f87d4a6f926edea263b450bc P 2260afde4ccb498189d7b25c00a2c727: Time spent starting tablet: real 0.045s user 0.005s sys 0.002s
I20240502 20:45:34.214236 1332 heartbeater.cc:503] Master 127.30.32.126:43021 was elected leader, sending a full tablet report...
I20240502 20:45:34.218307 746 catalog_manager.cc:5484] T e7080f04f87d4a6f926edea263b450bc P 2260afde4ccb498189d7b25c00a2c727 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2260afde4ccb498189d7b25c00a2c727 (127.30.32.68). New cstate: current_term: 1 leader_uuid: "2260afde4ccb498189d7b25c00a2c727" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2260afde4ccb498189d7b25c00a2c727" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 40723 } health_report { overall_health: HEALTHY } } }
I20240502 20:45:34.232713 30849 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 20:45:34.234212 30849 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 33be9a5b29214a8d84c2fd7a4936a920 to finish bootstrapping
I20240502 20:45:34.240342 30849 ts_itest-base.cc:233] Waiting for 0 tablets on tserver ed5783865eb74d39bee2a38e9139dea4 to finish bootstrapping
I20240502 20:45:34.246114 30849 ts_itest-base.cc:233] Waiting for 0 tablets on tserver f7fd7bf659f84ae184696b891db5221a to finish bootstrapping
I20240502 20:45:34.251811 30849 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 2260afde4ccb498189d7b25c00a2c727 to finish bootstrapping
I20240502 20:45:34.257380 30849 ts_itest-base.cc:233] Waiting for 0 tablets on tserver cec988e98db64e2f98b030efc16d4108 to finish bootstrapping
I20240502 20:45:34.263132 30849 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 6d231fa7f1684535af46a9b196111880 to finish bootstrapping
I20240502 20:45:34.268671 30849 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 0dfc68b63d15462398c76bfbff0e3cf8 to finish bootstrapping
I20240502 20:45:34.274372 30849 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskolF7xA/build/debug/bin/kudu with pid 1073
I20240502 20:45:34.280192 30849 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskolF7xA/build/debug/bin/kudu with pid 1204
I20240502 20:45:34.285713 30849 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskolF7xA/build/debug/bin/kudu with pid 1335
I20240502 20:45:34.291425 30849 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskolF7xA/build/debug/bin/kudu with pid 1466
I20240502 20:45:34.296820 30849 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskolF7xA/build/debug/bin/kudu with pid 1597
I20240502 20:45:38.053421 746 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53992:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:45:38.065310 1004 tablet_service.cc:1467] Processing CreateTablet for tablet 74b7ff29e276451985e7cbe72ceff148 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9738245a8d55447f94af91ee9ca2ce96]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:38.065311 873 tablet_service.cc:1467] Processing CreateTablet for tablet cf7de28bcb20418f99846c121cd88531 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9738245a8d55447f94af91ee9ca2ce96]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:38.065317 872 tablet_service.cc:1467] Processing CreateTablet for tablet 975b1d549985459da90b6cde03b0c5b8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9738245a8d55447f94af91ee9ca2ce96]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:38.066339 1004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74b7ff29e276451985e7cbe72ceff148. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:38.066339 872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 975b1d549985459da90b6cde03b0c5b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:38.066722 873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf7de28bcb20418f99846c121cd88531. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:38.071928 1762 tablet_bootstrap.cc:492] T 74b7ff29e276451985e7cbe72ceff148 P ed5783865eb74d39bee2a38e9139dea4: Bootstrap starting.
I20240502 20:45:38.071913 1761 tablet_bootstrap.cc:492] T cf7de28bcb20418f99846c121cd88531 P 33be9a5b29214a8d84c2fd7a4936a920: Bootstrap starting.
I20240502 20:45:38.074201 1762 tablet_bootstrap.cc:654] T 74b7ff29e276451985e7cbe72ceff148 P ed5783865eb74d39bee2a38e9139dea4: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:38.074223 1761 tablet_bootstrap.cc:654] T cf7de28bcb20418f99846c121cd88531 P 33be9a5b29214a8d84c2fd7a4936a920: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:38.075042 1762 log.cc:824] T 74b7ff29e276451985e7cbe72ceff148 P ed5783865eb74d39bee2a38e9139dea4: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:38.075042 1761 log.cc:824] T cf7de28bcb20418f99846c121cd88531 P 33be9a5b29214a8d84c2fd7a4936a920: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:38.076539 1762 tablet_bootstrap.cc:492] T 74b7ff29e276451985e7cbe72ceff148 P ed5783865eb74d39bee2a38e9139dea4: No bootstrap required, opened a new log
I20240502 20:45:38.076541 1761 tablet_bootstrap.cc:492] T cf7de28bcb20418f99846c121cd88531 P 33be9a5b29214a8d84c2fd7a4936a920: No bootstrap required, opened a new log
I20240502 20:45:38.076697 1761 ts_tablet_manager.cc:1367] T cf7de28bcb20418f99846c121cd88531 P 33be9a5b29214a8d84c2fd7a4936a920: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 20:45:38.076704 1762 ts_tablet_manager.cc:1367] T 74b7ff29e276451985e7cbe72ceff148 P ed5783865eb74d39bee2a38e9139dea4: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 20:45:38.079730 1762 raft_consensus.cc:348] T 74b7ff29e276451985e7cbe72ceff148 P ed5783865eb74d39bee2a38e9139dea4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } }
I20240502 20:45:38.079730 1761 raft_consensus.cc:348] T cf7de28bcb20418f99846c121cd88531 P 33be9a5b29214a8d84c2fd7a4936a920 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } }
I20240502 20:45:38.079895 1762 raft_consensus.cc:374] T 74b7ff29e276451985e7cbe72ceff148 P ed5783865eb74d39bee2a38e9139dea4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:38.079900 1761 raft_consensus.cc:374] T cf7de28bcb20418f99846c121cd88531 P 33be9a5b29214a8d84c2fd7a4936a920 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:38.079929 1762 raft_consensus.cc:729] T 74b7ff29e276451985e7cbe72ceff148 P ed5783865eb74d39bee2a38e9139dea4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed5783865eb74d39bee2a38e9139dea4, State: Initialized, Role: FOLLOWER
I20240502 20:45:38.079958 1761 raft_consensus.cc:729] T cf7de28bcb20418f99846c121cd88531 P 33be9a5b29214a8d84c2fd7a4936a920 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33be9a5b29214a8d84c2fd7a4936a920, State: Initialized, Role: FOLLOWER
I20240502 20:45:38.080310 1761 consensus_queue.cc:260] T cf7de28bcb20418f99846c121cd88531 P 33be9a5b29214a8d84c2fd7a4936a920 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } }
I20240502 20:45:38.080313 1762 consensus_queue.cc:260] T 74b7ff29e276451985e7cbe72ceff148 P ed5783865eb74d39bee2a38e9139dea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } }
I20240502 20:45:38.080422 1762 raft_consensus.cc:388] T 74b7ff29e276451985e7cbe72ceff148 P ed5783865eb74d39bee2a38e9139dea4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 20:45:38.080425 1761 raft_consensus.cc:388] T cf7de28bcb20418f99846c121cd88531 P 33be9a5b29214a8d84c2fd7a4936a920 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 20:45:38.080494 1762 raft_consensus.cc:482] T 74b7ff29e276451985e7cbe72ceff148 P ed5783865eb74d39bee2a38e9139dea4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 20:45:38.080494 1761 raft_consensus.cc:482] T cf7de28bcb20418f99846c121cd88531 P 33be9a5b29214a8d84c2fd7a4936a920 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 20:45:38.080595 1761 raft_consensus.cc:3037] T cf7de28bcb20418f99846c121cd88531 P 33be9a5b29214a8d84c2fd7a4936a920 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:38.080596 1762 raft_consensus.cc:3037] T 74b7ff29e276451985e7cbe72ceff148 P ed5783865eb74d39bee2a38e9139dea4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:38.081350 1762 raft_consensus.cc:504] T 74b7ff29e276451985e7cbe72ceff148 P ed5783865eb74d39bee2a38e9139dea4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } }
I20240502 20:45:38.081352 1761 raft_consensus.cc:504] T cf7de28bcb20418f99846c121cd88531 P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } }
I20240502 20:45:38.081637 1761 leader_election.cc:302] T cf7de28bcb20418f99846c121cd88531 P 33be9a5b29214a8d84c2fd7a4936a920 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33be9a5b29214a8d84c2fd7a4936a920; no voters:
I20240502 20:45:38.081645 1762 leader_election.cc:302] T 74b7ff29e276451985e7cbe72ceff148 P ed5783865eb74d39bee2a38e9139dea4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed5783865eb74d39bee2a38e9139dea4; no voters:
I20240502 20:45:38.081904 1761 leader_election.cc:288] T cf7de28bcb20418f99846c121cd88531 P 33be9a5b29214a8d84c2fd7a4936a920 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 20:45:38.081918 1762 leader_election.cc:288] T 74b7ff29e276451985e7cbe72ceff148 P ed5783865eb74d39bee2a38e9139dea4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 20:45:38.082005 1765 raft_consensus.cc:2781] T cf7de28bcb20418f99846c121cd88531 P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:38.082208 1765 raft_consensus.cc:686] T cf7de28bcb20418f99846c121cd88531 P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 LEADER]: Becoming Leader. State: Replica: 33be9a5b29214a8d84c2fd7a4936a920, State: Running, Role: LEADER
I20240502 20:45:38.082259 1768 raft_consensus.cc:2781] T 74b7ff29e276451985e7cbe72ceff148 P ed5783865eb74d39bee2a38e9139dea4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:38.082522 1768 raft_consensus.cc:686] T 74b7ff29e276451985e7cbe72ceff148 P ed5783865eb74d39bee2a38e9139dea4 [term 1 LEADER]: Becoming Leader. State: Replica: ed5783865eb74d39bee2a38e9139dea4, State: Running, Role: LEADER
I20240502 20:45:38.082502 1765 consensus_queue.cc:237] T cf7de28bcb20418f99846c121cd88531 P 33be9a5b29214a8d84c2fd7a4936a920 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } }
I20240502 20:45:38.082751 1761 ts_tablet_manager.cc:1398] T cf7de28bcb20418f99846c121cd88531 P 33be9a5b29214a8d84c2fd7a4936a920: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240502 20:45:38.082944 1762 ts_tablet_manager.cc:1398] T 74b7ff29e276451985e7cbe72ceff148 P ed5783865eb74d39bee2a38e9139dea4: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240502 20:45:38.083132 1761 tablet_bootstrap.cc:492] T 975b1d549985459da90b6cde03b0c5b8 P 33be9a5b29214a8d84c2fd7a4936a920: Bootstrap starting.
I20240502 20:45:38.083321 1768 consensus_queue.cc:237] T 74b7ff29e276451985e7cbe72ceff148 P ed5783865eb74d39bee2a38e9139dea4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } }
I20240502 20:45:38.084662 1761 tablet_bootstrap.cc:654] T 975b1d549985459da90b6cde03b0c5b8 P 33be9a5b29214a8d84c2fd7a4936a920: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:38.086153 1761 tablet_bootstrap.cc:492] T 975b1d549985459da90b6cde03b0c5b8 P 33be9a5b29214a8d84c2fd7a4936a920: No bootstrap required, opened a new log
I20240502 20:45:38.086268 1761 ts_tablet_manager.cc:1367] T 975b1d549985459da90b6cde03b0c5b8 P 33be9a5b29214a8d84c2fd7a4936a920: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:45:38.086452 746 catalog_manager.cc:5484] T cf7de28bcb20418f99846c121cd88531 P 33be9a5b29214a8d84c2fd7a4936a920 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33be9a5b29214a8d84c2fd7a4936a920 (127.30.32.65). New cstate: current_term: 1 leader_uuid: "33be9a5b29214a8d84c2fd7a4936a920" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } health_report { overall_health: HEALTHY } } }
I20240502 20:45:38.086817 1761 raft_consensus.cc:348] T 975b1d549985459da90b6cde03b0c5b8 P 33be9a5b29214a8d84c2fd7a4936a920 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } }
I20240502 20:45:38.086922 1761 raft_consensus.cc:374] T 975b1d549985459da90b6cde03b0c5b8 P 33be9a5b29214a8d84c2fd7a4936a920 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:38.086963 1761 raft_consensus.cc:729] T 975b1d549985459da90b6cde03b0c5b8 P 33be9a5b29214a8d84c2fd7a4936a920 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33be9a5b29214a8d84c2fd7a4936a920, State: Initialized, Role: FOLLOWER
I20240502 20:45:38.087107 1761 consensus_queue.cc:260] T 975b1d549985459da90b6cde03b0c5b8 P 33be9a5b29214a8d84c2fd7a4936a920 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } }
I20240502 20:45:38.087251 1761 raft_consensus.cc:388] T 975b1d549985459da90b6cde03b0c5b8 P 33be9a5b29214a8d84c2fd7a4936a920 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 20:45:38.087330 1761 raft_consensus.cc:482] T 975b1d549985459da90b6cde03b0c5b8 P 33be9a5b29214a8d84c2fd7a4936a920 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 20:45:38.087410 1761 raft_consensus.cc:3037] T 975b1d549985459da90b6cde03b0c5b8 P 33be9a5b29214a8d84c2fd7a4936a920 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:38.088088 751 catalog_manager.cc:5484] T 74b7ff29e276451985e7cbe72ceff148 P ed5783865eb74d39bee2a38e9139dea4 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed5783865eb74d39bee2a38e9139dea4 (127.30.32.66). New cstate: current_term: 1 leader_uuid: "ed5783865eb74d39bee2a38e9139dea4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } health_report { overall_health: HEALTHY } } }
I20240502 20:45:38.088348 1761 raft_consensus.cc:504] T 975b1d549985459da90b6cde03b0c5b8 P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } }
I20240502 20:45:38.088515 1761 leader_election.cc:302] T 975b1d549985459da90b6cde03b0c5b8 P 33be9a5b29214a8d84c2fd7a4936a920 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33be9a5b29214a8d84c2fd7a4936a920; no voters:
I20240502 20:45:38.088620 1761 leader_election.cc:288] T 975b1d549985459da90b6cde03b0c5b8 P 33be9a5b29214a8d84c2fd7a4936a920 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 20:45:38.088666 1765 raft_consensus.cc:2781] T 975b1d549985459da90b6cde03b0c5b8 P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:38.088760 1765 raft_consensus.cc:686] T 975b1d549985459da90b6cde03b0c5b8 P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 LEADER]: Becoming Leader. State: Replica: 33be9a5b29214a8d84c2fd7a4936a920, State: Running, Role: LEADER
I20240502 20:45:38.088867 1765 consensus_queue.cc:237] T 975b1d549985459da90b6cde03b0c5b8 P 33be9a5b29214a8d84c2fd7a4936a920 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } }
I20240502 20:45:38.089017 1761 ts_tablet_manager.cc:1398] T 975b1d549985459da90b6cde03b0c5b8 P 33be9a5b29214a8d84c2fd7a4936a920: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:45:38.090373 746 catalog_manager.cc:5484] T 975b1d549985459da90b6cde03b0c5b8 P 33be9a5b29214a8d84c2fd7a4936a920 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33be9a5b29214a8d84c2fd7a4936a920 (127.30.32.65). New cstate: current_term: 1 leader_uuid: "33be9a5b29214a8d84c2fd7a4936a920" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } health_report { overall_health: HEALTHY } } }
W20240502 20:45:38.204598 1777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:38.204838 1777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:38.207850 1777 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 84.1851 ms
time per row: 0.0420925 ms
I20240502 20:45:38.331522 743 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53992:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:45:38.336802 1003 tablet_service.cc:1467] Processing CreateTablet for tablet 7a888b6096c2427f8a7f67b26c3c00b7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=da321cf3a7944f5ba716ef84b816582e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:38.336915 873 tablet_service.cc:1467] Processing CreateTablet for tablet ff0d71af3fd4483caa41325903f9ac39 (DEFAULT_TABLE table=rebalance_test_table_1 [id=da321cf3a7944f5ba716ef84b816582e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:38.337158 1003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a888b6096c2427f8a7f67b26c3c00b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:38.337220 873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff0d71af3fd4483caa41325903f9ac39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:38.338841 1761 tablet_bootstrap.cc:492] T ff0d71af3fd4483caa41325903f9ac39 P 33be9a5b29214a8d84c2fd7a4936a920: Bootstrap starting.
I20240502 20:45:38.338867 1762 tablet_bootstrap.cc:492] T 7a888b6096c2427f8a7f67b26c3c00b7 P ed5783865eb74d39bee2a38e9139dea4: Bootstrap starting.
I20240502 20:45:38.336779 1004 tablet_service.cc:1467] Processing CreateTablet for tablet a6c9b839b6a346aca42a280aed623913 (DEFAULT_TABLE table=rebalance_test_table_1 [id=da321cf3a7944f5ba716ef84b816582e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:38.339251 1004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6c9b839b6a346aca42a280aed623913. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:38.339701 1761 tablet_bootstrap.cc:654] T ff0d71af3fd4483caa41325903f9ac39 P 33be9a5b29214a8d84c2fd7a4936a920: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:38.339859 1762 tablet_bootstrap.cc:654] T 7a888b6096c2427f8a7f67b26c3c00b7 P ed5783865eb74d39bee2a38e9139dea4: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:38.340801 1761 tablet_bootstrap.cc:492] T ff0d71af3fd4483caa41325903f9ac39 P 33be9a5b29214a8d84c2fd7a4936a920: No bootstrap required, opened a new log
I20240502 20:45:38.340894 1761 ts_tablet_manager.cc:1367] T ff0d71af3fd4483caa41325903f9ac39 P 33be9a5b29214a8d84c2fd7a4936a920: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:45:38.340955 1762 tablet_bootstrap.cc:492] T 7a888b6096c2427f8a7f67b26c3c00b7 P ed5783865eb74d39bee2a38e9139dea4: No bootstrap required, opened a new log
I20240502 20:45:38.341063 1762 ts_tablet_manager.cc:1367] T 7a888b6096c2427f8a7f67b26c3c00b7 P ed5783865eb74d39bee2a38e9139dea4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:38.341392 1761 raft_consensus.cc:348] T ff0d71af3fd4483caa41325903f9ac39 P 33be9a5b29214a8d84c2fd7a4936a920 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } }
I20240502 20:45:38.341480 1761 raft_consensus.cc:374] T ff0d71af3fd4483caa41325903f9ac39 P 33be9a5b29214a8d84c2fd7a4936a920 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:38.341507 1761 raft_consensus.cc:729] T ff0d71af3fd4483caa41325903f9ac39 P 33be9a5b29214a8d84c2fd7a4936a920 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33be9a5b29214a8d84c2fd7a4936a920, State: Initialized, Role: FOLLOWER
I20240502 20:45:38.341598 1762 raft_consensus.cc:348] T 7a888b6096c2427f8a7f67b26c3c00b7 P ed5783865eb74d39bee2a38e9139dea4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } }
I20240502 20:45:38.341635 1761 consensus_queue.cc:260] T ff0d71af3fd4483caa41325903f9ac39 P 33be9a5b29214a8d84c2fd7a4936a920 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } }
I20240502 20:45:38.341675 1762 raft_consensus.cc:374] T 7a888b6096c2427f8a7f67b26c3c00b7 P ed5783865eb74d39bee2a38e9139dea4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:38.341706 1762 raft_consensus.cc:729] T 7a888b6096c2427f8a7f67b26c3c00b7 P ed5783865eb74d39bee2a38e9139dea4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed5783865eb74d39bee2a38e9139dea4, State: Initialized, Role: FOLLOWER
I20240502 20:45:38.341714 1761 raft_consensus.cc:388] T ff0d71af3fd4483caa41325903f9ac39 P 33be9a5b29214a8d84c2fd7a4936a920 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 20:45:38.341763 1761 raft_consensus.cc:482] T ff0d71af3fd4483caa41325903f9ac39 P 33be9a5b29214a8d84c2fd7a4936a920 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 20:45:38.341807 1761 raft_consensus.cc:3037] T ff0d71af3fd4483caa41325903f9ac39 P 33be9a5b29214a8d84c2fd7a4936a920 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:38.341831 1762 consensus_queue.cc:260] T 7a888b6096c2427f8a7f67b26c3c00b7 P ed5783865eb74d39bee2a38e9139dea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } }
I20240502 20:45:38.341907 1762 raft_consensus.cc:388] T 7a888b6096c2427f8a7f67b26c3c00b7 P ed5783865eb74d39bee2a38e9139dea4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 20:45:38.341948 1762 raft_consensus.cc:482] T 7a888b6096c2427f8a7f67b26c3c00b7 P ed5783865eb74d39bee2a38e9139dea4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 20:45:38.341984 1762 raft_consensus.cc:3037] T 7a888b6096c2427f8a7f67b26c3c00b7 P ed5783865eb74d39bee2a38e9139dea4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:38.342649 1761 raft_consensus.cc:504] T ff0d71af3fd4483caa41325903f9ac39 P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } }
I20240502 20:45:38.342799 1761 leader_election.cc:302] T ff0d71af3fd4483caa41325903f9ac39 P 33be9a5b29214a8d84c2fd7a4936a920 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33be9a5b29214a8d84c2fd7a4936a920; no voters:
I20240502 20:45:38.342815 1762 raft_consensus.cc:504] T 7a888b6096c2427f8a7f67b26c3c00b7 P ed5783865eb74d39bee2a38e9139dea4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } }
I20240502 20:45:38.342902 1761 leader_election.cc:288] T ff0d71af3fd4483caa41325903f9ac39 P 33be9a5b29214a8d84c2fd7a4936a920 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 20:45:38.342919 1762 leader_election.cc:302] T 7a888b6096c2427f8a7f67b26c3c00b7 P ed5783865eb74d39bee2a38e9139dea4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed5783865eb74d39bee2a38e9139dea4; no voters:
I20240502 20:45:38.342958 1765 raft_consensus.cc:2781] T ff0d71af3fd4483caa41325903f9ac39 P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:38.342996 1762 leader_election.cc:288] T 7a888b6096c2427f8a7f67b26c3c00b7 P ed5783865eb74d39bee2a38e9139dea4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 20:45:38.343055 1765 raft_consensus.cc:686] T ff0d71af3fd4483caa41325903f9ac39 P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 LEADER]: Becoming Leader. State: Replica: 33be9a5b29214a8d84c2fd7a4936a920, State: Running, Role: LEADER
I20240502 20:45:38.343163 1770 raft_consensus.cc:2781] T 7a888b6096c2427f8a7f67b26c3c00b7 P ed5783865eb74d39bee2a38e9139dea4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:38.343178 1765 consensus_queue.cc:237] T ff0d71af3fd4483caa41325903f9ac39 P 33be9a5b29214a8d84c2fd7a4936a920 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } }
I20240502 20:45:38.343252 1770 raft_consensus.cc:686] T 7a888b6096c2427f8a7f67b26c3c00b7 P ed5783865eb74d39bee2a38e9139dea4 [term 1 LEADER]: Becoming Leader. State: Replica: ed5783865eb74d39bee2a38e9139dea4, State: Running, Role: LEADER
I20240502 20:45:38.343384 1761 ts_tablet_manager.cc:1398] T ff0d71af3fd4483caa41325903f9ac39 P 33be9a5b29214a8d84c2fd7a4936a920: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 20:45:38.343398 1770 consensus_queue.cc:237] T 7a888b6096c2427f8a7f67b26c3c00b7 P ed5783865eb74d39bee2a38e9139dea4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } }
I20240502 20:45:38.343174 1762 ts_tablet_manager.cc:1398] T 7a888b6096c2427f8a7f67b26c3c00b7 P ed5783865eb74d39bee2a38e9139dea4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 20:45:38.343777 1762 tablet_bootstrap.cc:492] T a6c9b839b6a346aca42a280aed623913 P ed5783865eb74d39bee2a38e9139dea4: Bootstrap starting.
I20240502 20:45:38.344627 1762 tablet_bootstrap.cc:654] T a6c9b839b6a346aca42a280aed623913 P ed5783865eb74d39bee2a38e9139dea4: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:38.345065 743 catalog_manager.cc:5484] T 7a888b6096c2427f8a7f67b26c3c00b7 P ed5783865eb74d39bee2a38e9139dea4 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed5783865eb74d39bee2a38e9139dea4 (127.30.32.66). New cstate: current_term: 1 leader_uuid: "ed5783865eb74d39bee2a38e9139dea4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } health_report { overall_health: HEALTHY } } }
I20240502 20:45:38.345324 744 catalog_manager.cc:5484] T ff0d71af3fd4483caa41325903f9ac39 P 33be9a5b29214a8d84c2fd7a4936a920 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33be9a5b29214a8d84c2fd7a4936a920 (127.30.32.65). New cstate: current_term: 1 leader_uuid: "33be9a5b29214a8d84c2fd7a4936a920" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } health_report { overall_health: HEALTHY } } }
I20240502 20:45:38.345659 1762 tablet_bootstrap.cc:492] T a6c9b839b6a346aca42a280aed623913 P ed5783865eb74d39bee2a38e9139dea4: No bootstrap required, opened a new log
I20240502 20:45:38.345774 1762 ts_tablet_manager.cc:1367] T a6c9b839b6a346aca42a280aed623913 P ed5783865eb74d39bee2a38e9139dea4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 20:45:38.346202 1762 raft_consensus.cc:348] T a6c9b839b6a346aca42a280aed623913 P ed5783865eb74d39bee2a38e9139dea4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } }
I20240502 20:45:38.346307 1762 raft_consensus.cc:374] T a6c9b839b6a346aca42a280aed623913 P ed5783865eb74d39bee2a38e9139dea4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:38.346347 1762 raft_consensus.cc:729] T a6c9b839b6a346aca42a280aed623913 P ed5783865eb74d39bee2a38e9139dea4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed5783865eb74d39bee2a38e9139dea4, State: Initialized, Role: FOLLOWER
I20240502 20:45:38.346462 1762 consensus_queue.cc:260] T a6c9b839b6a346aca42a280aed623913 P ed5783865eb74d39bee2a38e9139dea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } }
I20240502 20:45:38.346549 1762 raft_consensus.cc:388] T a6c9b839b6a346aca42a280aed623913 P ed5783865eb74d39bee2a38e9139dea4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 20:45:38.346593 1762 raft_consensus.cc:482] T a6c9b839b6a346aca42a280aed623913 P ed5783865eb74d39bee2a38e9139dea4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 20:45:38.346642 1762 raft_consensus.cc:3037] T a6c9b839b6a346aca42a280aed623913 P ed5783865eb74d39bee2a38e9139dea4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:38.347376 1762 raft_consensus.cc:504] T a6c9b839b6a346aca42a280aed623913 P ed5783865eb74d39bee2a38e9139dea4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } }
I20240502 20:45:38.347553 1762 leader_election.cc:302] T a6c9b839b6a346aca42a280aed623913 P ed5783865eb74d39bee2a38e9139dea4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed5783865eb74d39bee2a38e9139dea4; no voters:
I20240502 20:45:38.347698 1770 raft_consensus.cc:2781] T a6c9b839b6a346aca42a280aed623913 P ed5783865eb74d39bee2a38e9139dea4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:38.347772 1770 raft_consensus.cc:686] T a6c9b839b6a346aca42a280aed623913 P ed5783865eb74d39bee2a38e9139dea4 [term 1 LEADER]: Becoming Leader. State: Replica: ed5783865eb74d39bee2a38e9139dea4, State: Running, Role: LEADER
I20240502 20:45:38.347880 1770 consensus_queue.cc:237] T a6c9b839b6a346aca42a280aed623913 P ed5783865eb74d39bee2a38e9139dea4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } }
I20240502 20:45:38.348227 1762 leader_election.cc:288] T a6c9b839b6a346aca42a280aed623913 P ed5783865eb74d39bee2a38e9139dea4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 20:45:38.348490 1762 ts_tablet_manager.cc:1398] T a6c9b839b6a346aca42a280aed623913 P ed5783865eb74d39bee2a38e9139dea4: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 20:45:38.349189 744 catalog_manager.cc:5484] T a6c9b839b6a346aca42a280aed623913 P ed5783865eb74d39bee2a38e9139dea4 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed5783865eb74d39bee2a38e9139dea4 (127.30.32.66). New cstate: current_term: 1 leader_uuid: "ed5783865eb74d39bee2a38e9139dea4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } health_report { overall_health: HEALTHY } } }
W20240502 20:45:38.461357 1819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:38.461589 1819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:38.464519 1819 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
INSERT report
rows total: 2000
time total: 59.9866 ms
time per row: 0.0299933 ms
I20240502 20:45:38.563208 745 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53992:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:45:38.568507 873 tablet_service.cc:1467] Processing CreateTablet for tablet 49eb89bf68d942398c68e2bf1d0f6370 (DEFAULT_TABLE table=rebalance_test_table_2 [id=fddf24b281f54ee394d7a819baadce47]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:38.568572 1004 tablet_service.cc:1467] Processing CreateTablet for tablet 8d06cc46e1b04a809b88c070996ba1ed (DEFAULT_TABLE table=rebalance_test_table_2 [id=fddf24b281f54ee394d7a819baadce47]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:38.568842 873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49eb89bf68d942398c68e2bf1d0f6370. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:38.568816 872 tablet_service.cc:1467] Processing CreateTablet for tablet 21a27f13d4104a8a87aa8b935c6c8dfa (DEFAULT_TABLE table=rebalance_test_table_2 [id=fddf24b281f54ee394d7a819baadce47]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:38.568881 1004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d06cc46e1b04a809b88c070996ba1ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:38.569227 872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21a27f13d4104a8a87aa8b935c6c8dfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:38.570709 1762 tablet_bootstrap.cc:492] T 8d06cc46e1b04a809b88c070996ba1ed P ed5783865eb74d39bee2a38e9139dea4: Bootstrap starting.
I20240502 20:45:38.571077 1761 tablet_bootstrap.cc:492] T 49eb89bf68d942398c68e2bf1d0f6370 P 33be9a5b29214a8d84c2fd7a4936a920: Bootstrap starting.
I20240502 20:45:38.571586 1762 tablet_bootstrap.cc:654] T 8d06cc46e1b04a809b88c070996ba1ed P ed5783865eb74d39bee2a38e9139dea4: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:38.571923 1761 tablet_bootstrap.cc:654] T 49eb89bf68d942398c68e2bf1d0f6370 P 33be9a5b29214a8d84c2fd7a4936a920: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:38.572713 1762 tablet_bootstrap.cc:492] T 8d06cc46e1b04a809b88c070996ba1ed P ed5783865eb74d39bee2a38e9139dea4: No bootstrap required, opened a new log
I20240502 20:45:38.572791 1762 ts_tablet_manager.cc:1367] T 8d06cc46e1b04a809b88c070996ba1ed P ed5783865eb74d39bee2a38e9139dea4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:45:38.573091 1761 tablet_bootstrap.cc:492] T 49eb89bf68d942398c68e2bf1d0f6370 P 33be9a5b29214a8d84c2fd7a4936a920: No bootstrap required, opened a new log
I20240502 20:45:38.573160 1761 ts_tablet_manager.cc:1367] T 49eb89bf68d942398c68e2bf1d0f6370 P 33be9a5b29214a8d84c2fd7a4936a920: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:38.573241 1762 raft_consensus.cc:348] T 8d06cc46e1b04a809b88c070996ba1ed P ed5783865eb74d39bee2a38e9139dea4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } }
I20240502 20:45:38.573338 1762 raft_consensus.cc:374] T 8d06cc46e1b04a809b88c070996ba1ed P ed5783865eb74d39bee2a38e9139dea4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:38.573381 1762 raft_consensus.cc:729] T 8d06cc46e1b04a809b88c070996ba1ed P ed5783865eb74d39bee2a38e9139dea4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed5783865eb74d39bee2a38e9139dea4, State: Initialized, Role: FOLLOWER
I20240502 20:45:38.573515 1762 consensus_queue.cc:260] T 8d06cc46e1b04a809b88c070996ba1ed P ed5783865eb74d39bee2a38e9139dea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } }
I20240502 20:45:38.573596 1762 raft_consensus.cc:388] T 8d06cc46e1b04a809b88c070996ba1ed P ed5783865eb74d39bee2a38e9139dea4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 20:45:38.573565 1761 raft_consensus.cc:348] T 49eb89bf68d942398c68e2bf1d0f6370 P 33be9a5b29214a8d84c2fd7a4936a920 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } }
I20240502 20:45:38.573637 1762 raft_consensus.cc:482] T 8d06cc46e1b04a809b88c070996ba1ed P ed5783865eb74d39bee2a38e9139dea4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 20:45:38.573640 1761 raft_consensus.cc:374] T 49eb89bf68d942398c68e2bf1d0f6370 P 33be9a5b29214a8d84c2fd7a4936a920 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:38.573679 1761 raft_consensus.cc:729] T 49eb89bf68d942398c68e2bf1d0f6370 P 33be9a5b29214a8d84c2fd7a4936a920 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33be9a5b29214a8d84c2fd7a4936a920, State: Initialized, Role: FOLLOWER
I20240502 20:45:38.573691 1762 raft_consensus.cc:3037] T 8d06cc46e1b04a809b88c070996ba1ed P ed5783865eb74d39bee2a38e9139dea4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:38.573812 1761 consensus_queue.cc:260] T 49eb89bf68d942398c68e2bf1d0f6370 P 33be9a5b29214a8d84c2fd7a4936a920 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } }
I20240502 20:45:38.573911 1761 raft_consensus.cc:388] T 49eb89bf68d942398c68e2bf1d0f6370 P 33be9a5b29214a8d84c2fd7a4936a920 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 20:45:38.573951 1761 raft_consensus.cc:482] T 49eb89bf68d942398c68e2bf1d0f6370 P 33be9a5b29214a8d84c2fd7a4936a920 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 20:45:38.573994 1761 raft_consensus.cc:3037] T 49eb89bf68d942398c68e2bf1d0f6370 P 33be9a5b29214a8d84c2fd7a4936a920 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:38.574499 1762 raft_consensus.cc:504] T 8d06cc46e1b04a809b88c070996ba1ed P ed5783865eb74d39bee2a38e9139dea4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } }
I20240502 20:45:38.574606 1762 leader_election.cc:302] T 8d06cc46e1b04a809b88c070996ba1ed P ed5783865eb74d39bee2a38e9139dea4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed5783865eb74d39bee2a38e9139dea4; no voters:
I20240502 20:45:38.574677 1762 leader_election.cc:288] T 8d06cc46e1b04a809b88c070996ba1ed P ed5783865eb74d39bee2a38e9139dea4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 20:45:38.574738 1770 raft_consensus.cc:2781] T 8d06cc46e1b04a809b88c070996ba1ed P ed5783865eb74d39bee2a38e9139dea4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:38.574810 1770 raft_consensus.cc:686] T 8d06cc46e1b04a809b88c070996ba1ed P ed5783865eb74d39bee2a38e9139dea4 [term 1 LEADER]: Becoming Leader. State: Replica: ed5783865eb74d39bee2a38e9139dea4, State: Running, Role: LEADER
I20240502 20:45:38.574738 1761 raft_consensus.cc:504] T 49eb89bf68d942398c68e2bf1d0f6370 P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } }
I20240502 20:45:38.574906 1770 consensus_queue.cc:237] T 8d06cc46e1b04a809b88c070996ba1ed P ed5783865eb74d39bee2a38e9139dea4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } }
I20240502 20:45:38.575021 1761 leader_election.cc:302] T 49eb89bf68d942398c68e2bf1d0f6370 P 33be9a5b29214a8d84c2fd7a4936a920 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33be9a5b29214a8d84c2fd7a4936a920; no voters:
I20240502 20:45:38.575100 1761 leader_election.cc:288] T 49eb89bf68d942398c68e2bf1d0f6370 P 33be9a5b29214a8d84c2fd7a4936a920 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 20:45:38.575139 1762 ts_tablet_manager.cc:1398] T 8d06cc46e1b04a809b88c070996ba1ed P ed5783865eb74d39bee2a38e9139dea4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 20:45:38.575172 1767 raft_consensus.cc:2781] T 49eb89bf68d942398c68e2bf1d0f6370 P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:38.575248 1767 raft_consensus.cc:686] T 49eb89bf68d942398c68e2bf1d0f6370 P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 LEADER]: Becoming Leader. State: Replica: 33be9a5b29214a8d84c2fd7a4936a920, State: Running, Role: LEADER
I20240502 20:45:38.575376 1767 consensus_queue.cc:237] T 49eb89bf68d942398c68e2bf1d0f6370 P 33be9a5b29214a8d84c2fd7a4936a920 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } }
I20240502 20:45:38.575552 1761 ts_tablet_manager.cc:1398] T 49eb89bf68d942398c68e2bf1d0f6370 P 33be9a5b29214a8d84c2fd7a4936a920: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:38.575711 1761 tablet_bootstrap.cc:492] T 21a27f13d4104a8a87aa8b935c6c8dfa P 33be9a5b29214a8d84c2fd7a4936a920: Bootstrap starting.
I20240502 20:45:38.576170 745 catalog_manager.cc:5484] T 8d06cc46e1b04a809b88c070996ba1ed P ed5783865eb74d39bee2a38e9139dea4 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed5783865eb74d39bee2a38e9139dea4 (127.30.32.66). New cstate: current_term: 1 leader_uuid: "ed5783865eb74d39bee2a38e9139dea4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } health_report { overall_health: HEALTHY } } }
I20240502 20:45:38.576587 1761 tablet_bootstrap.cc:654] T 21a27f13d4104a8a87aa8b935c6c8dfa P 33be9a5b29214a8d84c2fd7a4936a920: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:38.576735 749 catalog_manager.cc:5484] T 49eb89bf68d942398c68e2bf1d0f6370 P 33be9a5b29214a8d84c2fd7a4936a920 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33be9a5b29214a8d84c2fd7a4936a920 (127.30.32.65). New cstate: current_term: 1 leader_uuid: "33be9a5b29214a8d84c2fd7a4936a920" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } health_report { overall_health: HEALTHY } } }
I20240502 20:45:38.577991 1761 tablet_bootstrap.cc:492] T 21a27f13d4104a8a87aa8b935c6c8dfa P 33be9a5b29214a8d84c2fd7a4936a920: No bootstrap required, opened a new log
I20240502 20:45:38.578073 1761 ts_tablet_manager.cc:1367] T 21a27f13d4104a8a87aa8b935c6c8dfa P 33be9a5b29214a8d84c2fd7a4936a920: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:45:38.578500 1761 raft_consensus.cc:348] T 21a27f13d4104a8a87aa8b935c6c8dfa P 33be9a5b29214a8d84c2fd7a4936a920 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } }
I20240502 20:45:38.578588 1761 raft_consensus.cc:374] T 21a27f13d4104a8a87aa8b935c6c8dfa P 33be9a5b29214a8d84c2fd7a4936a920 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:38.578619 1761 raft_consensus.cc:729] T 21a27f13d4104a8a87aa8b935c6c8dfa P 33be9a5b29214a8d84c2fd7a4936a920 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33be9a5b29214a8d84c2fd7a4936a920, State: Initialized, Role: FOLLOWER
I20240502 20:45:38.578753 1761 consensus_queue.cc:260] T 21a27f13d4104a8a87aa8b935c6c8dfa P 33be9a5b29214a8d84c2fd7a4936a920 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } }
I20240502 20:45:38.578811 1761 raft_consensus.cc:388] T 21a27f13d4104a8a87aa8b935c6c8dfa P 33be9a5b29214a8d84c2fd7a4936a920 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 20:45:38.578833 1761 raft_consensus.cc:482] T 21a27f13d4104a8a87aa8b935c6c8dfa P 33be9a5b29214a8d84c2fd7a4936a920 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 20:45:38.578860 1761 raft_consensus.cc:3037] T 21a27f13d4104a8a87aa8b935c6c8dfa P 33be9a5b29214a8d84c2fd7a4936a920 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:38.579667 1761 raft_consensus.cc:504] T 21a27f13d4104a8a87aa8b935c6c8dfa P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } }
I20240502 20:45:38.579782 1761 leader_election.cc:302] T 21a27f13d4104a8a87aa8b935c6c8dfa P 33be9a5b29214a8d84c2fd7a4936a920 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33be9a5b29214a8d84c2fd7a4936a920; no voters:
I20240502 20:45:38.579865 1761 leader_election.cc:288] T 21a27f13d4104a8a87aa8b935c6c8dfa P 33be9a5b29214a8d84c2fd7a4936a920 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 20:45:38.579916 1765 raft_consensus.cc:2781] T 21a27f13d4104a8a87aa8b935c6c8dfa P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:38.579982 1765 raft_consensus.cc:686] T 21a27f13d4104a8a87aa8b935c6c8dfa P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 LEADER]: Becoming Leader. State: Replica: 33be9a5b29214a8d84c2fd7a4936a920, State: Running, Role: LEADER
I20240502 20:45:38.580068 1761 ts_tablet_manager.cc:1398] T 21a27f13d4104a8a87aa8b935c6c8dfa P 33be9a5b29214a8d84c2fd7a4936a920: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 20:45:38.580080 1765 consensus_queue.cc:237] T 21a27f13d4104a8a87aa8b935c6c8dfa P 33be9a5b29214a8d84c2fd7a4936a920 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } }
I20240502 20:45:38.582652 745 catalog_manager.cc:5484] T 21a27f13d4104a8a87aa8b935c6c8dfa P 33be9a5b29214a8d84c2fd7a4936a920 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33be9a5b29214a8d84c2fd7a4936a920 (127.30.32.65). New cstate: current_term: 1 leader_uuid: "33be9a5b29214a8d84c2fd7a4936a920" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } health_report { overall_health: HEALTHY } } }
W20240502 20:45:38.692149 1854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:38.692445 1854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:38.695684 1854 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
INSERT report
rows total: 2000
time total: 77.1098 ms
time per row: 0.0385549 ms
I20240502 20:45:38.814402 749 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53992:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:45:38.819886 1004 tablet_service.cc:1467] Processing CreateTablet for tablet afc220634a164930bdaa8ece818fb2e7 (DEFAULT_TABLE table=rebalance_test_table_3 [id=2a2a74b349514a8aa4c6c02c2533a51f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:38.819933 1003 tablet_service.cc:1467] Processing CreateTablet for tablet 89fc553dd07e4af5bd37801a34c4ac66 (DEFAULT_TABLE table=rebalance_test_table_3 [id=2a2a74b349514a8aa4c6c02c2533a51f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:38.819988 872 tablet_service.cc:1467] Processing CreateTablet for tablet da05217d82a74bd6aeb54330c3d6d370 (DEFAULT_TABLE table=rebalance_test_table_3 [id=2a2a74b349514a8aa4c6c02c2533a51f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:38.820314 1004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afc220634a164930bdaa8ece818fb2e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:38.820359 872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da05217d82a74bd6aeb54330c3d6d370. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:38.820461 1003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89fc553dd07e4af5bd37801a34c4ac66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:38.822520 1761 tablet_bootstrap.cc:492] T da05217d82a74bd6aeb54330c3d6d370 P 33be9a5b29214a8d84c2fd7a4936a920: Bootstrap starting.
I20240502 20:45:38.822520 1762 tablet_bootstrap.cc:492] T afc220634a164930bdaa8ece818fb2e7 P ed5783865eb74d39bee2a38e9139dea4: Bootstrap starting.
I20240502 20:45:38.823400 1761 tablet_bootstrap.cc:654] T da05217d82a74bd6aeb54330c3d6d370 P 33be9a5b29214a8d84c2fd7a4936a920: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:38.823400 1762 tablet_bootstrap.cc:654] T afc220634a164930bdaa8ece818fb2e7 P ed5783865eb74d39bee2a38e9139dea4: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:38.824448 1761 tablet_bootstrap.cc:492] T da05217d82a74bd6aeb54330c3d6d370 P 33be9a5b29214a8d84c2fd7a4936a920: No bootstrap required, opened a new log
I20240502 20:45:38.824472 1762 tablet_bootstrap.cc:492] T afc220634a164930bdaa8ece818fb2e7 P ed5783865eb74d39bee2a38e9139dea4: No bootstrap required, opened a new log
I20240502 20:45:38.824520 1761 ts_tablet_manager.cc:1367] T da05217d82a74bd6aeb54330c3d6d370 P 33be9a5b29214a8d84c2fd7a4936a920: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 20:45:38.824554 1762 ts_tablet_manager.cc:1367] T afc220634a164930bdaa8ece818fb2e7 P ed5783865eb74d39bee2a38e9139dea4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:45:38.824901 1761 raft_consensus.cc:348] T da05217d82a74bd6aeb54330c3d6d370 P 33be9a5b29214a8d84c2fd7a4936a920 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } }
I20240502 20:45:38.824923 1762 raft_consensus.cc:348] T afc220634a164930bdaa8ece818fb2e7 P ed5783865eb74d39bee2a38e9139dea4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } }
I20240502 20:45:38.825017 1761 raft_consensus.cc:374] T da05217d82a74bd6aeb54330c3d6d370 P 33be9a5b29214a8d84c2fd7a4936a920 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:38.825033 1762 raft_consensus.cc:374] T afc220634a164930bdaa8ece818fb2e7 P ed5783865eb74d39bee2a38e9139dea4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:38.825057 1761 raft_consensus.cc:729] T da05217d82a74bd6aeb54330c3d6d370 P 33be9a5b29214a8d84c2fd7a4936a920 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33be9a5b29214a8d84c2fd7a4936a920, State: Initialized, Role: FOLLOWER
I20240502 20:45:38.825086 1762 raft_consensus.cc:729] T afc220634a164930bdaa8ece818fb2e7 P ed5783865eb74d39bee2a38e9139dea4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed5783865eb74d39bee2a38e9139dea4, State: Initialized, Role: FOLLOWER
I20240502 20:45:38.825181 1761 consensus_queue.cc:260] T da05217d82a74bd6aeb54330c3d6d370 P 33be9a5b29214a8d84c2fd7a4936a920 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } }
I20240502 20:45:38.825277 1761 raft_consensus.cc:388] T da05217d82a74bd6aeb54330c3d6d370 P 33be9a5b29214a8d84c2fd7a4936a920 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 20:45:38.825248 1762 consensus_queue.cc:260] T afc220634a164930bdaa8ece818fb2e7 P ed5783865eb74d39bee2a38e9139dea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } }
I20240502 20:45:38.825317 1761 raft_consensus.cc:482] T da05217d82a74bd6aeb54330c3d6d370 P 33be9a5b29214a8d84c2fd7a4936a920 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 20:45:38.825335 1762 raft_consensus.cc:388] T afc220634a164930bdaa8ece818fb2e7 P ed5783865eb74d39bee2a38e9139dea4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 20:45:38.825364 1761 raft_consensus.cc:3037] T da05217d82a74bd6aeb54330c3d6d370 P 33be9a5b29214a8d84c2fd7a4936a920 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:38.825374 1762 raft_consensus.cc:482] T afc220634a164930bdaa8ece818fb2e7 P ed5783865eb74d39bee2a38e9139dea4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 20:45:38.825420 1762 raft_consensus.cc:3037] T afc220634a164930bdaa8ece818fb2e7 P ed5783865eb74d39bee2a38e9139dea4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:38.826023 1761 raft_consensus.cc:504] T da05217d82a74bd6aeb54330c3d6d370 P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } }
I20240502 20:45:38.826027 1762 raft_consensus.cc:504] T afc220634a164930bdaa8ece818fb2e7 P ed5783865eb74d39bee2a38e9139dea4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } }
I20240502 20:45:38.826131 1761 leader_election.cc:302] T da05217d82a74bd6aeb54330c3d6d370 P 33be9a5b29214a8d84c2fd7a4936a920 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33be9a5b29214a8d84c2fd7a4936a920; no voters:
I20240502 20:45:38.826159 1762 leader_election.cc:302] T afc220634a164930bdaa8ece818fb2e7 P ed5783865eb74d39bee2a38e9139dea4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed5783865eb74d39bee2a38e9139dea4; no voters:
I20240502 20:45:38.826251 1762 leader_election.cc:288] T afc220634a164930bdaa8ece818fb2e7 P ed5783865eb74d39bee2a38e9139dea4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 20:45:38.826282 1765 raft_consensus.cc:2781] T da05217d82a74bd6aeb54330c3d6d370 P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:38.826293 1770 raft_consensus.cc:2781] T afc220634a164930bdaa8ece818fb2e7 P ed5783865eb74d39bee2a38e9139dea4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:38.826355 1765 raft_consensus.cc:686] T da05217d82a74bd6aeb54330c3d6d370 P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 LEADER]: Becoming Leader. State: Replica: 33be9a5b29214a8d84c2fd7a4936a920, State: Running, Role: LEADER
I20240502 20:45:38.826478 1765 consensus_queue.cc:237] T da05217d82a74bd6aeb54330c3d6d370 P 33be9a5b29214a8d84c2fd7a4936a920 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } }
I20240502 20:45:38.826695 1770 raft_consensus.cc:686] T afc220634a164930bdaa8ece818fb2e7 P ed5783865eb74d39bee2a38e9139dea4 [term 1 LEADER]: Becoming Leader. State: Replica: ed5783865eb74d39bee2a38e9139dea4, State: Running, Role: LEADER
I20240502 20:45:38.826815 1770 consensus_queue.cc:237] T afc220634a164930bdaa8ece818fb2e7 P ed5783865eb74d39bee2a38e9139dea4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } }
I20240502 20:45:38.826938 1761 leader_election.cc:288] T da05217d82a74bd6aeb54330c3d6d370 P 33be9a5b29214a8d84c2fd7a4936a920 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 20:45:38.826972 1762 ts_tablet_manager.cc:1398] T afc220634a164930bdaa8ece818fb2e7 P ed5783865eb74d39bee2a38e9139dea4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:38.827203 1761 ts_tablet_manager.cc:1398] T da05217d82a74bd6aeb54330c3d6d370 P 33be9a5b29214a8d84c2fd7a4936a920: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 20:45:38.827409 1762 tablet_bootstrap.cc:492] T 89fc553dd07e4af5bd37801a34c4ac66 P ed5783865eb74d39bee2a38e9139dea4: Bootstrap starting.
I20240502 20:45:38.828338 1762 tablet_bootstrap.cc:654] T 89fc553dd07e4af5bd37801a34c4ac66 P ed5783865eb74d39bee2a38e9139dea4: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:38.828302 744 catalog_manager.cc:5484] T da05217d82a74bd6aeb54330c3d6d370 P 33be9a5b29214a8d84c2fd7a4936a920 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33be9a5b29214a8d84c2fd7a4936a920 (127.30.32.65). New cstate: current_term: 1 leader_uuid: "33be9a5b29214a8d84c2fd7a4936a920" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } health_report { overall_health: HEALTHY } } }
I20240502 20:45:38.828323 749 catalog_manager.cc:5484] T afc220634a164930bdaa8ece818fb2e7 P ed5783865eb74d39bee2a38e9139dea4 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed5783865eb74d39bee2a38e9139dea4 (127.30.32.66). New cstate: current_term: 1 leader_uuid: "ed5783865eb74d39bee2a38e9139dea4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } health_report { overall_health: HEALTHY } } }
I20240502 20:45:38.829465 1762 tablet_bootstrap.cc:492] T 89fc553dd07e4af5bd37801a34c4ac66 P ed5783865eb74d39bee2a38e9139dea4: No bootstrap required, opened a new log
I20240502 20:45:38.829543 1762 ts_tablet_manager.cc:1367] T 89fc553dd07e4af5bd37801a34c4ac66 P ed5783865eb74d39bee2a38e9139dea4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:38.829936 1762 raft_consensus.cc:348] T 89fc553dd07e4af5bd37801a34c4ac66 P ed5783865eb74d39bee2a38e9139dea4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } }
I20240502 20:45:38.830020 1762 raft_consensus.cc:374] T 89fc553dd07e4af5bd37801a34c4ac66 P ed5783865eb74d39bee2a38e9139dea4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:38.830058 1762 raft_consensus.cc:729] T 89fc553dd07e4af5bd37801a34c4ac66 P ed5783865eb74d39bee2a38e9139dea4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed5783865eb74d39bee2a38e9139dea4, State: Initialized, Role: FOLLOWER
I20240502 20:45:38.830164 1762 consensus_queue.cc:260] T 89fc553dd07e4af5bd37801a34c4ac66 P ed5783865eb74d39bee2a38e9139dea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } }
I20240502 20:45:38.830236 1762 raft_consensus.cc:388] T 89fc553dd07e4af5bd37801a34c4ac66 P ed5783865eb74d39bee2a38e9139dea4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 20:45:38.830274 1762 raft_consensus.cc:482] T 89fc553dd07e4af5bd37801a34c4ac66 P ed5783865eb74d39bee2a38e9139dea4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 20:45:38.830312 1762 raft_consensus.cc:3037] T 89fc553dd07e4af5bd37801a34c4ac66 P ed5783865eb74d39bee2a38e9139dea4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:38.830945 1762 raft_consensus.cc:504] T 89fc553dd07e4af5bd37801a34c4ac66 P ed5783865eb74d39bee2a38e9139dea4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } }
I20240502 20:45:38.831033 1762 leader_election.cc:302] T 89fc553dd07e4af5bd37801a34c4ac66 P ed5783865eb74d39bee2a38e9139dea4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed5783865eb74d39bee2a38e9139dea4; no voters:
I20240502 20:45:38.831100 1762 leader_election.cc:288] T 89fc553dd07e4af5bd37801a34c4ac66 P ed5783865eb74d39bee2a38e9139dea4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 20:45:38.831241 1766 raft_consensus.cc:2781] T 89fc553dd07e4af5bd37801a34c4ac66 P ed5783865eb74d39bee2a38e9139dea4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:38.831313 1762 ts_tablet_manager.cc:1398] T 89fc553dd07e4af5bd37801a34c4ac66 P ed5783865eb74d39bee2a38e9139dea4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:45:38.831321 1766 raft_consensus.cc:686] T 89fc553dd07e4af5bd37801a34c4ac66 P ed5783865eb74d39bee2a38e9139dea4 [term 1 LEADER]: Becoming Leader. State: Replica: ed5783865eb74d39bee2a38e9139dea4, State: Running, Role: LEADER
I20240502 20:45:38.831447 1766 consensus_queue.cc:237] T 89fc553dd07e4af5bd37801a34c4ac66 P ed5783865eb74d39bee2a38e9139dea4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } }
I20240502 20:45:38.832898 744 catalog_manager.cc:5484] T 89fc553dd07e4af5bd37801a34c4ac66 P ed5783865eb74d39bee2a38e9139dea4 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed5783865eb74d39bee2a38e9139dea4 (127.30.32.66). New cstate: current_term: 1 leader_uuid: "ed5783865eb74d39bee2a38e9139dea4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } health_report { overall_health: HEALTHY } } }
W20240502 20:45:38.938385 1885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:38.938623 1885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:38.941609 1885 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 63.2937 ms
time per row: 0.0316468 ms
I20240502 20:45:39.046273 744 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53992:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:45:39.051703 873 tablet_service.cc:1467] Processing CreateTablet for tablet 63e2878c2981429cbcb41837f3f0c44a (DEFAULT_TABLE table=rebalance_test_table_4 [id=1c8687d7588245508f9c47ea918ebc6a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:39.051759 1003 tablet_service.cc:1467] Processing CreateTablet for tablet 83593d702ac542c28bd1721338a55c5c (DEFAULT_TABLE table=rebalance_test_table_4 [id=1c8687d7588245508f9c47ea918ebc6a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:39.051703 872 tablet_service.cc:1467] Processing CreateTablet for tablet 0734f8b2eeb04ad99ff827c0ecf90663 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1c8687d7588245508f9c47ea918ebc6a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:39.052068 1003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83593d702ac542c28bd1721338a55c5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:39.052078 873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63e2878c2981429cbcb41837f3f0c44a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:39.052299 872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0734f8b2eeb04ad99ff827c0ecf90663. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:39.053788 1762 tablet_bootstrap.cc:492] T 83593d702ac542c28bd1721338a55c5c P ed5783865eb74d39bee2a38e9139dea4: Bootstrap starting.
I20240502 20:45:39.054273 1761 tablet_bootstrap.cc:492] T 0734f8b2eeb04ad99ff827c0ecf90663 P 33be9a5b29214a8d84c2fd7a4936a920: Bootstrap starting.
I20240502 20:45:39.054623 1762 tablet_bootstrap.cc:654] T 83593d702ac542c28bd1721338a55c5c P ed5783865eb74d39bee2a38e9139dea4: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:39.055105 1761 tablet_bootstrap.cc:654] T 0734f8b2eeb04ad99ff827c0ecf90663 P 33be9a5b29214a8d84c2fd7a4936a920: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:39.055629 1762 tablet_bootstrap.cc:492] T 83593d702ac542c28bd1721338a55c5c P ed5783865eb74d39bee2a38e9139dea4: No bootstrap required, opened a new log
I20240502 20:45:39.055701 1762 ts_tablet_manager.cc:1367] T 83593d702ac542c28bd1721338a55c5c P ed5783865eb74d39bee2a38e9139dea4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:45:39.056093 1761 tablet_bootstrap.cc:492] T 0734f8b2eeb04ad99ff827c0ecf90663 P 33be9a5b29214a8d84c2fd7a4936a920: No bootstrap required, opened a new log
I20240502 20:45:39.056118 1762 raft_consensus.cc:348] T 83593d702ac542c28bd1721338a55c5c P ed5783865eb74d39bee2a38e9139dea4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } }
I20240502 20:45:39.056187 1761 ts_tablet_manager.cc:1367] T 0734f8b2eeb04ad99ff827c0ecf90663 P 33be9a5b29214a8d84c2fd7a4936a920: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 20:45:39.056232 1762 raft_consensus.cc:374] T 83593d702ac542c28bd1721338a55c5c P ed5783865eb74d39bee2a38e9139dea4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:39.056272 1762 raft_consensus.cc:729] T 83593d702ac542c28bd1721338a55c5c P ed5783865eb74d39bee2a38e9139dea4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed5783865eb74d39bee2a38e9139dea4, State: Initialized, Role: FOLLOWER
I20240502 20:45:39.056397 1762 consensus_queue.cc:260] T 83593d702ac542c28bd1721338a55c5c P ed5783865eb74d39bee2a38e9139dea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } }
I20240502 20:45:39.056488 1762 raft_consensus.cc:388] T 83593d702ac542c28bd1721338a55c5c P ed5783865eb74d39bee2a38e9139dea4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 20:45:39.056524 1762 raft_consensus.cc:482] T 83593d702ac542c28bd1721338a55c5c P ed5783865eb74d39bee2a38e9139dea4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 20:45:39.056562 1762 raft_consensus.cc:3037] T 83593d702ac542c28bd1721338a55c5c P ed5783865eb74d39bee2a38e9139dea4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:39.056596 1761 raft_consensus.cc:348] T 0734f8b2eeb04ad99ff827c0ecf90663 P 33be9a5b29214a8d84c2fd7a4936a920 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } }
I20240502 20:45:39.056705 1761 raft_consensus.cc:374] T 0734f8b2eeb04ad99ff827c0ecf90663 P 33be9a5b29214a8d84c2fd7a4936a920 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:39.056746 1761 raft_consensus.cc:729] T 0734f8b2eeb04ad99ff827c0ecf90663 P 33be9a5b29214a8d84c2fd7a4936a920 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33be9a5b29214a8d84c2fd7a4936a920, State: Initialized, Role: FOLLOWER
I20240502 20:45:39.056869 1761 consensus_queue.cc:260] T 0734f8b2eeb04ad99ff827c0ecf90663 P 33be9a5b29214a8d84c2fd7a4936a920 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } }
I20240502 20:45:39.056946 1761 raft_consensus.cc:388] T 0734f8b2eeb04ad99ff827c0ecf90663 P 33be9a5b29214a8d84c2fd7a4936a920 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 20:45:39.057001 1761 raft_consensus.cc:482] T 0734f8b2eeb04ad99ff827c0ecf90663 P 33be9a5b29214a8d84c2fd7a4936a920 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 20:45:39.057042 1761 raft_consensus.cc:3037] T 0734f8b2eeb04ad99ff827c0ecf90663 P 33be9a5b29214a8d84c2fd7a4936a920 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:39.057531 1762 raft_consensus.cc:504] T 83593d702ac542c28bd1721338a55c5c P ed5783865eb74d39bee2a38e9139dea4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } }
I20240502 20:45:39.057648 1762 leader_election.cc:302] T 83593d702ac542c28bd1721338a55c5c P ed5783865eb74d39bee2a38e9139dea4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed5783865eb74d39bee2a38e9139dea4; no voters:
I20240502 20:45:39.057744 1762 leader_election.cc:288] T 83593d702ac542c28bd1721338a55c5c P ed5783865eb74d39bee2a38e9139dea4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 20:45:39.057811 1768 raft_consensus.cc:2781] T 83593d702ac542c28bd1721338a55c5c P ed5783865eb74d39bee2a38e9139dea4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:39.057914 1768 raft_consensus.cc:686] T 83593d702ac542c28bd1721338a55c5c P ed5783865eb74d39bee2a38e9139dea4 [term 1 LEADER]: Becoming Leader. State: Replica: ed5783865eb74d39bee2a38e9139dea4, State: Running, Role: LEADER
I20240502 20:45:39.057921 1761 raft_consensus.cc:504] T 0734f8b2eeb04ad99ff827c0ecf90663 P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } }
I20240502 20:45:39.057988 1762 ts_tablet_manager.cc:1398] T 83593d702ac542c28bd1721338a55c5c P ed5783865eb74d39bee2a38e9139dea4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:39.058037 1761 leader_election.cc:302] T 0734f8b2eeb04ad99ff827c0ecf90663 P 33be9a5b29214a8d84c2fd7a4936a920 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33be9a5b29214a8d84c2fd7a4936a920; no voters:
I20240502 20:45:39.058017 1768 consensus_queue.cc:237] T 83593d702ac542c28bd1721338a55c5c P ed5783865eb74d39bee2a38e9139dea4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } }
I20240502 20:45:39.058127 1761 leader_election.cc:288] T 0734f8b2eeb04ad99ff827c0ecf90663 P 33be9a5b29214a8d84c2fd7a4936a920 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 20:45:39.058180 1767 raft_consensus.cc:2781] T 0734f8b2eeb04ad99ff827c0ecf90663 P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:39.058247 1767 raft_consensus.cc:686] T 0734f8b2eeb04ad99ff827c0ecf90663 P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 LEADER]: Becoming Leader. State: Replica: 33be9a5b29214a8d84c2fd7a4936a920, State: Running, Role: LEADER
I20240502 20:45:39.058352 1767 consensus_queue.cc:237] T 0734f8b2eeb04ad99ff827c0ecf90663 P 33be9a5b29214a8d84c2fd7a4936a920 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } }
I20240502 20:45:39.058367 1761 ts_tablet_manager.cc:1398] T 0734f8b2eeb04ad99ff827c0ecf90663 P 33be9a5b29214a8d84c2fd7a4936a920: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240502 20:45:39.059376 1761 tablet_bootstrap.cc:492] T 63e2878c2981429cbcb41837f3f0c44a P 33be9a5b29214a8d84c2fd7a4936a920: Bootstrap starting.
I20240502 20:45:39.059396 744 catalog_manager.cc:5484] T 83593d702ac542c28bd1721338a55c5c P ed5783865eb74d39bee2a38e9139dea4 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed5783865eb74d39bee2a38e9139dea4 (127.30.32.66). New cstate: current_term: 1 leader_uuid: "ed5783865eb74d39bee2a38e9139dea4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } health_report { overall_health: HEALTHY } } }
I20240502 20:45:39.059836 750 catalog_manager.cc:5484] T 0734f8b2eeb04ad99ff827c0ecf90663 P 33be9a5b29214a8d84c2fd7a4936a920 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33be9a5b29214a8d84c2fd7a4936a920 (127.30.32.65). New cstate: current_term: 1 leader_uuid: "33be9a5b29214a8d84c2fd7a4936a920" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } health_report { overall_health: HEALTHY } } }
I20240502 20:45:39.060508 1761 tablet_bootstrap.cc:654] T 63e2878c2981429cbcb41837f3f0c44a P 33be9a5b29214a8d84c2fd7a4936a920: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:39.061664 1761 tablet_bootstrap.cc:492] T 63e2878c2981429cbcb41837f3f0c44a P 33be9a5b29214a8d84c2fd7a4936a920: No bootstrap required, opened a new log
I20240502 20:45:39.061769 1761 ts_tablet_manager.cc:1367] T 63e2878c2981429cbcb41837f3f0c44a P 33be9a5b29214a8d84c2fd7a4936a920: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 20:45:39.062745 1761 raft_consensus.cc:348] T 63e2878c2981429cbcb41837f3f0c44a P 33be9a5b29214a8d84c2fd7a4936a920 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } }
I20240502 20:45:39.062857 1761 raft_consensus.cc:374] T 63e2878c2981429cbcb41837f3f0c44a P 33be9a5b29214a8d84c2fd7a4936a920 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:39.062927 1761 raft_consensus.cc:729] T 63e2878c2981429cbcb41837f3f0c44a P 33be9a5b29214a8d84c2fd7a4936a920 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33be9a5b29214a8d84c2fd7a4936a920, State: Initialized, Role: FOLLOWER
I20240502 20:45:39.063067 1761 consensus_queue.cc:260] T 63e2878c2981429cbcb41837f3f0c44a P 33be9a5b29214a8d84c2fd7a4936a920 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } }
I20240502 20:45:39.063163 1761 raft_consensus.cc:388] T 63e2878c2981429cbcb41837f3f0c44a P 33be9a5b29214a8d84c2fd7a4936a920 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 20:45:39.063205 1761 raft_consensus.cc:482] T 63e2878c2981429cbcb41837f3f0c44a P 33be9a5b29214a8d84c2fd7a4936a920 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 20:45:39.063244 1761 raft_consensus.cc:3037] T 63e2878c2981429cbcb41837f3f0c44a P 33be9a5b29214a8d84c2fd7a4936a920 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:39.064020 1761 raft_consensus.cc:504] T 63e2878c2981429cbcb41837f3f0c44a P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } }
I20240502 20:45:39.064139 1761 leader_election.cc:302] T 63e2878c2981429cbcb41837f3f0c44a P 33be9a5b29214a8d84c2fd7a4936a920 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33be9a5b29214a8d84c2fd7a4936a920; no voters:
I20240502 20:45:39.064267 1761 leader_election.cc:288] T 63e2878c2981429cbcb41837f3f0c44a P 33be9a5b29214a8d84c2fd7a4936a920 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 20:45:39.064330 1769 raft_consensus.cc:2781] T 63e2878c2981429cbcb41837f3f0c44a P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:39.064427 1769 raft_consensus.cc:686] T 63e2878c2981429cbcb41837f3f0c44a P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 LEADER]: Becoming Leader. State: Replica: 33be9a5b29214a8d84c2fd7a4936a920, State: Running, Role: LEADER
I20240502 20:45:39.064512 1761 ts_tablet_manager.cc:1398] T 63e2878c2981429cbcb41837f3f0c44a P 33be9a5b29214a8d84c2fd7a4936a920: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240502 20:45:39.064543 1769 consensus_queue.cc:237] T 63e2878c2981429cbcb41837f3f0c44a P 33be9a5b29214a8d84c2fd7a4936a920 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } }
I20240502 20:45:39.065807 744 catalog_manager.cc:5484] T 63e2878c2981429cbcb41837f3f0c44a P 33be9a5b29214a8d84c2fd7a4936a920 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33be9a5b29214a8d84c2fd7a4936a920 (127.30.32.65). New cstate: current_term: 1 leader_uuid: "33be9a5b29214a8d84c2fd7a4936a920" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } health_report { overall_health: HEALTHY } } }
W20240502 20:45:39.178357 1917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:39.178752 1917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:39.181840 1917 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 63.2161 ms
time per row: 0.031608 ms
I20240502 20:45:39.287006 30849 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskolF7xA/build/debug/bin/kudu
/tmp/dist-test-taskolF7xA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.32.67:42407
--local_ip_for_outbound_sockets=127.30.32.67
--tserver_master_addrs=127.30.32.126:43021
--webserver_port=41483
--webserver_interface=127.30.32.67
--builtin_ntp_servers=127.30.32.84:35991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 20:45:39.395018 1943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:39.395255 1943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:39.395334 1943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:39.398499 1943 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:39.398576 1943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:39.398679 1943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.32.67
I20240502 20:45:39.402419 1943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.32.84:35991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.32.67:42407
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.30.32.67
--webserver_port=41483
--tserver_master_addrs=127.30.32.126:43021
--never_fsync=true
--heap_profile_path=/tmp/kudu.1943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.32.67
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:39.403368 1943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:39.404512 1943 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:39.413251 1951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:39.413996 1949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:39.414703 1943 server_base.cc:1034] running on GCE node
W20240502 20:45:39.415215 1948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:39.415596 1943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:39.416168 1943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:39.417320 1943 hybrid_clock.cc:648] HybridClock initialized: now 1714682739417307 us; error 26 us; skew 500 ppm
I20240502 20:45:39.419219 1943 webserver.cc:460] Webserver started at http://127.30.32.67:41483/ using document root <none> and password file <none>
I20240502 20:45:39.419823 1943 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:39.419907 1943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:39.423525 1943 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:39.425607 1957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:39.426645 1943 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:39.426776 1943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "f7fd7bf659f84ae184696b891db5221a"
format_stamp: "Formatted at 2024-05-02 20:45:33 on dist-test-slave-6336"
I20240502 20:45:39.427214 1943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:39.445825 1943 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:39.446465 1943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:39.446630 1943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:39.447168 1943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:39.448045 1943 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:39.448107 1943 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:39.448168 1943 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:39.448196 1943 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:39.456699 1943 rpc_server.cc:307] RPC server started. Bound to: 127.30.32.67:42407
I20240502 20:45:39.456732 2070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.32.67:42407 every 8 connection(s)
I20240502 20:45:39.457777 1943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 20:45:39.462734 30849 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskolF7xA/build/debug/bin/kudu as pid 1943
I20240502 20:45:39.463124 30849 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskolF7xA/build/debug/bin/kudu
/tmp/dist-test-taskolF7xA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.32.68:40723
--local_ip_for_outbound_sockets=127.30.32.68
--tserver_master_addrs=127.30.32.126:43021
--webserver_port=42261
--webserver_interface=127.30.32.68
--builtin_ntp_servers=127.30.32.84:35991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 20:45:39.466985 2071 heartbeater.cc:347] Connected to a master server at 127.30.32.126:43021
I20240502 20:45:39.467195 2071 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:39.467692 2071 heartbeater.cc:511] Master 127.30.32.126:43021 requested a full tablet report, sending...
I20240502 20:45:39.468959 743 ts_manager.cc:194] Re-registered known tserver with Master: f7fd7bf659f84ae184696b891db5221a (127.30.32.67:42407)
I20240502 20:45:39.469895 743 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.32.67:35105
W20240502 20:45:39.573154 2074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:39.573390 2074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:39.573437 2074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:39.576489 2074 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:39.576558 2074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:39.576628 2074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.32.68
I20240502 20:45:39.580296 2074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.32.84:35991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.32.68:40723
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.30.32.68
--webserver_port=42261
--tserver_master_addrs=127.30.32.126:43021
--never_fsync=true
--heap_profile_path=/tmp/kudu.2074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.32.68
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:39.581218 2074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:39.582317 2074 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:39.590775 2080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:39.591305 2079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:39.591953 2082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:39.592696 2074 server_base.cc:1034] running on GCE node
I20240502 20:45:39.593070 2074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:39.593575 2074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:39.594735 2074 hybrid_clock.cc:648] HybridClock initialized: now 1714682739594723 us; error 26 us; skew 500 ppm
I20240502 20:45:39.596459 2074 webserver.cc:460] Webserver started at http://127.30.32.68:42261/ using document root <none> and password file <none>
I20240502 20:45:39.597025 2074 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:39.597108 2074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:39.600332 2074 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:39.603626 2088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:39.604933 2074 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240502 20:45:39.605072 2074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "2260afde4ccb498189d7b25c00a2c727"
format_stamp: "Formatted at 2024-05-02 20:45:33 on dist-test-slave-6336"
I20240502 20:45:39.605602 2074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:39.617439 2074 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:39.618048 2074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:39.618204 2074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:39.618717 2074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:39.619825 2095 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240502 20:45:39.622908 2074 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240502 20:45:39.622960 2074 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20240502 20:45:39.623013 2074 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240502 20:45:39.624387 2074 ts_tablet_manager.cc:590] Registered 1 tablets
I20240502 20:45:39.624428 2074 ts_tablet_manager.cc:569] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20240502 20:45:39.624665 2095 tablet_bootstrap.cc:492] T e7080f04f87d4a6f926edea263b450bc P 2260afde4ccb498189d7b25c00a2c727: Bootstrap starting.
I20240502 20:45:39.635680 2074 rpc_server.cc:307] RPC server started. Bound to: 127.30.32.68:40723
I20240502 20:45:39.635761 2202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.32.68:40723 every 8 connection(s)
I20240502 20:45:39.637154 2074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 20:45:39.639442 30849 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskolF7xA/build/debug/bin/kudu as pid 2074
I20240502 20:45:39.639684 2095 log.cc:824] T e7080f04f87d4a6f926edea263b450bc P 2260afde4ccb498189d7b25c00a2c727: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:39.639855 30849 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskolF7xA/build/debug/bin/kudu
/tmp/dist-test-taskolF7xA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.32.69:34789
--local_ip_for_outbound_sockets=127.30.32.69
--tserver_master_addrs=127.30.32.126:43021
--webserver_port=43257
--webserver_interface=127.30.32.69
--builtin_ntp_servers=127.30.32.84:35991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 20:45:39.642433 2095 tablet_bootstrap.cc:492] T e7080f04f87d4a6f926edea263b450bc P 2260afde4ccb498189d7b25c00a2c727: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:45:39.642753 2095 tablet_bootstrap.cc:492] T e7080f04f87d4a6f926edea263b450bc P 2260afde4ccb498189d7b25c00a2c727: Bootstrap complete.
I20240502 20:45:39.643945 2095 ts_tablet_manager.cc:1367] T e7080f04f87d4a6f926edea263b450bc P 2260afde4ccb498189d7b25c00a2c727: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.008s
I20240502 20:45:39.646760 2203 heartbeater.cc:347] Connected to a master server at 127.30.32.126:43021
I20240502 20:45:39.646976 2203 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:39.647576 2203 heartbeater.cc:511] Master 127.30.32.126:43021 requested a full tablet report, sending...
I20240502 20:45:39.647344 2095 raft_consensus.cc:348] T e7080f04f87d4a6f926edea263b450bc P 2260afde4ccb498189d7b25c00a2c727 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2260afde4ccb498189d7b25c00a2c727" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 40723 } }
I20240502 20:45:39.647734 2095 raft_consensus.cc:729] T e7080f04f87d4a6f926edea263b450bc P 2260afde4ccb498189d7b25c00a2c727 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2260afde4ccb498189d7b25c00a2c727, State: Initialized, Role: FOLLOWER
I20240502 20:45:39.648226 2095 consensus_queue.cc:260] T e7080f04f87d4a6f926edea263b450bc P 2260afde4ccb498189d7b25c00a2c727 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2260afde4ccb498189d7b25c00a2c727" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 40723 } }
I20240502 20:45:39.648401 2095 raft_consensus.cc:388] T e7080f04f87d4a6f926edea263b450bc P 2260afde4ccb498189d7b25c00a2c727 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 20:45:39.648509 2095 raft_consensus.cc:482] T e7080f04f87d4a6f926edea263b450bc P 2260afde4ccb498189d7b25c00a2c727 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 20:45:39.648620 2095 raft_consensus.cc:3037] T e7080f04f87d4a6f926edea263b450bc P 2260afde4ccb498189d7b25c00a2c727 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:39.650301 743 ts_manager.cc:194] Re-registered known tserver with Master: 2260afde4ccb498189d7b25c00a2c727 (127.30.32.68:40723)
I20240502 20:45:39.650386 2095 raft_consensus.cc:504] T e7080f04f87d4a6f926edea263b450bc P 2260afde4ccb498189d7b25c00a2c727 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2260afde4ccb498189d7b25c00a2c727" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 40723 } }
I20240502 20:45:39.650651 2095 leader_election.cc:302] T e7080f04f87d4a6f926edea263b450bc P 2260afde4ccb498189d7b25c00a2c727 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2260afde4ccb498189d7b25c00a2c727; no voters:
I20240502 20:45:39.650902 2095 leader_election.cc:288] T e7080f04f87d4a6f926edea263b450bc P 2260afde4ccb498189d7b25c00a2c727 [CANDIDATE]: Term 2 election: Requested vote from peers
I20240502 20:45:39.650995 2208 raft_consensus.cc:2781] T e7080f04f87d4a6f926edea263b450bc P 2260afde4ccb498189d7b25c00a2c727 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:45:39.651147 2208 raft_consensus.cc:686] T e7080f04f87d4a6f926edea263b450bc P 2260afde4ccb498189d7b25c00a2c727 [term 2 LEADER]: Becoming Leader. State: Replica: 2260afde4ccb498189d7b25c00a2c727, State: Running, Role: LEADER
I20240502 20:45:39.651417 743 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.32.68:33683
I20240502 20:45:39.651412 2208 consensus_queue.cc:237] T e7080f04f87d4a6f926edea263b450bc P 2260afde4ccb498189d7b25c00a2c727 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2260afde4ccb498189d7b25c00a2c727" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 40723 } }
I20240502 20:45:39.651674 2095 ts_tablet_manager.cc:1398] T e7080f04f87d4a6f926edea263b450bc P 2260afde4ccb498189d7b25c00a2c727: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20240502 20:45:39.653856 2203 heartbeater.cc:503] Master 127.30.32.126:43021 was elected leader, sending a full tablet report...
I20240502 20:45:39.654983 743 catalog_manager.cc:5484] T e7080f04f87d4a6f926edea263b450bc P 2260afde4ccb498189d7b25c00a2c727 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "2260afde4ccb498189d7b25c00a2c727" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2260afde4ccb498189d7b25c00a2c727" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 40723 } health_report { overall_health: HEALTHY } } }
W20240502 20:45:39.754036 2207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:39.754242 2207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:39.754287 2207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:39.757304 2207 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:39.757364 2207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:39.757457 2207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.32.69
I20240502 20:45:39.760927 2207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.32.84:35991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.32.69:34789
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.30.32.69
--webserver_port=43257
--tserver_master_addrs=127.30.32.126:43021
--never_fsync=true
--heap_profile_path=/tmp/kudu.2207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.32.69
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:39.761796 2207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:39.762910 2207 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:39.771216 2220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:39.771813 2221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:39.772429 2223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:39.773170 2207 server_base.cc:1034] running on GCE node
I20240502 20:45:39.773510 2207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:39.774041 2207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:39.775200 2207 hybrid_clock.cc:648] HybridClock initialized: now 1714682739775184 us; error 27 us; skew 500 ppm
I20240502 20:45:39.777024 2207 webserver.cc:460] Webserver started at http://127.30.32.69:43257/ using document root <none> and password file <none>
I20240502 20:45:39.777593 2207 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:39.777664 2207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:39.781001 2207 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 20:45:39.782850 2229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:39.783815 2207 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:39.783929 2207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "cec988e98db64e2f98b030efc16d4108"
format_stamp: "Formatted at 2024-05-02 20:45:33 on dist-test-slave-6336"
I20240502 20:45:39.784363 2207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:39.816339 2207 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:39.817111 2207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:39.817279 2207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:39.817855 2207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:39.818790 2207 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:39.818842 2207 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:39.818904 2207 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:39.818931 2207 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:39.827661 2207 rpc_server.cc:307] RPC server started. Bound to: 127.30.32.69:34789
I20240502 20:45:39.827695 2342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.32.69:34789 every 8 connection(s)
I20240502 20:45:39.828711 2207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 20:45:39.836195 30849 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskolF7xA/build/debug/bin/kudu as pid 2207
I20240502 20:45:39.836652 30849 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskolF7xA/build/debug/bin/kudu
/tmp/dist-test-taskolF7xA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.32.70:43291
--local_ip_for_outbound_sockets=127.30.32.70
--tserver_master_addrs=127.30.32.126:43021
--webserver_port=41997
--webserver_interface=127.30.32.70
--builtin_ntp_servers=127.30.32.84:35991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 20:45:39.837690 2343 heartbeater.cc:347] Connected to a master server at 127.30.32.126:43021
I20240502 20:45:39.837913 2343 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:39.838600 2343 heartbeater.cc:511] Master 127.30.32.126:43021 requested a full tablet report, sending...
I20240502 20:45:39.839726 743 ts_manager.cc:194] Re-registered known tserver with Master: cec988e98db64e2f98b030efc16d4108 (127.30.32.69:34789)
I20240502 20:45:39.840323 743 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.32.69:45533
W20240502 20:45:39.947499 2346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:39.947705 2346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:39.947762 2346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:39.950865 2346 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:39.950929 2346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:39.951025 2346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.32.70
I20240502 20:45:39.954622 2346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.32.84:35991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.32.70:43291
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.30.32.70
--webserver_port=41997
--tserver_master_addrs=127.30.32.126:43021
--never_fsync=true
--heap_profile_path=/tmp/kudu.2346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.32.70
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:39.955452 2346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:39.956563 2346 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:39.964624 2354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:39.965189 2352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:39.965860 2351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:39.966586 2346 server_base.cc:1034] running on GCE node
I20240502 20:45:39.966943 2346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:39.967459 2346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:39.968629 2346 hybrid_clock.cc:648] HybridClock initialized: now 1714682739968619 us; error 23 us; skew 500 ppm
I20240502 20:45:39.970646 2346 webserver.cc:460] Webserver started at http://127.30.32.70:41997/ using document root <none> and password file <none>
I20240502 20:45:39.971223 2346 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:39.971305 2346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:39.974895 2346 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20240502 20:45:39.976670 2360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:39.977713 2346 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 20:45:39.977820 2346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "6d231fa7f1684535af46a9b196111880"
format_stamp: "Formatted at 2024-05-02 20:45:33 on dist-test-slave-6336"
I20240502 20:45:39.978219 2346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:40.016531 2346 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:40.017257 2346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:40.017416 2346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:40.017967 2346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:40.018889 2346 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:40.018947 2346 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:40.019006 2346 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:40.019033 2346 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:40.027284 2346 rpc_server.cc:307] RPC server started. Bound to: 127.30.32.70:43291
I20240502 20:45:40.027320 2473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.32.70:43291 every 8 connection(s)
I20240502 20:45:40.028277 2346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 20:45:40.032895 30849 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskolF7xA/build/debug/bin/kudu as pid 2346
I20240502 20:45:40.033311 30849 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskolF7xA/build/debug/bin/kudu
/tmp/dist-test-taskolF7xA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.32.71:40439
--local_ip_for_outbound_sockets=127.30.32.71
--tserver_master_addrs=127.30.32.126:43021
--webserver_port=42625
--webserver_interface=127.30.32.71
--builtin_ntp_servers=127.30.32.84:35991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 20:45:40.037904 2474 heartbeater.cc:347] Connected to a master server at 127.30.32.126:43021
I20240502 20:45:40.038115 2474 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:40.038609 2474 heartbeater.cc:511] Master 127.30.32.126:43021 requested a full tablet report, sending...
I20240502 20:45:40.039677 743 ts_manager.cc:194] Re-registered known tserver with Master: 6d231fa7f1684535af46a9b196111880 (127.30.32.70:43291)
I20240502 20:45:40.040529 743 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.32.70:35375
W20240502 20:45:40.144299 2477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:40.144505 2477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:40.144546 2477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:40.147542 2477 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:40.147612 2477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:40.147678 2477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.32.71
I20240502 20:45:40.151324 2477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.32.84:35991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.32.71:40439
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.30.32.71
--webserver_port=42625
--tserver_master_addrs=127.30.32.126:43021
--never_fsync=true
--heap_profile_path=/tmp/kudu.2477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.32.71
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:40.152145 2477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:40.153299 2477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:40.161736 2483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:40.162305 2485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:40.162983 2482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:40.163741 2477 server_base.cc:1034] running on GCE node
I20240502 20:45:40.164148 2477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:40.164714 2477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:40.165869 2477 hybrid_clock.cc:648] HybridClock initialized: now 1714682740165833 us; error 54 us; skew 500 ppm
I20240502 20:45:40.167673 2477 webserver.cc:460] Webserver started at http://127.30.32.71:42625/ using document root <none> and password file <none>
I20240502 20:45:40.168238 2477 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:40.168318 2477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:40.171901 2477 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:40.173864 2491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:40.174863 2477 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240502 20:45:40.174968 2477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "0dfc68b63d15462398c76bfbff0e3cf8"
format_stamp: "Formatted at 2024-05-02 20:45:34 on dist-test-slave-6336"
I20240502 20:45:40.175372 2477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:40.219282 2477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:40.220054 2477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:40.220222 2477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:40.220772 2477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:40.221690 2477 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:40.221737 2477 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:40.221777 2477 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:40.221799 2477 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:40.230589 2477 rpc_server.cc:307] RPC server started. Bound to: 127.30.32.71:40439
I20240502 20:45:40.230645 2604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.32.71:40439 every 8 connection(s)
I20240502 20:45:40.231696 2477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 20:45:40.239897 30849 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskolF7xA/build/debug/bin/kudu as pid 2477
I20240502 20:45:40.240185 2605 heartbeater.cc:347] Connected to a master server at 127.30.32.126:43021
I20240502 20:45:40.240417 2605 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:40.241076 2605 heartbeater.cc:511] Master 127.30.32.126:43021 requested a full tablet report, sending...
I20240502 20:45:40.242170 743 ts_manager.cc:194] Re-registered known tserver with Master: 0dfc68b63d15462398c76bfbff0e3cf8 (127.30.32.71:40439)
I20240502 20:45:40.242713 743 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.32.71:39273
I20240502 20:45:40.435173 873 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:45:40.436071 2137 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:45:40.436437 1003 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:45:40.437438 2539 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:40.440045 2277 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:40.451308 2005 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:40.454094 2408 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:40.472048 2071 heartbeater.cc:503] Master 127.30.32.126:43021 was elected leader, sending a full tablet report...
I20240502 20:45:40.579023 2539 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:40.583885 873 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:45:40.585558 2005 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:40.586598 2137 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:45:40.587163 2277 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:40.593204 1003 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:45:40.597627 2408 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:40.640832 893 consensus_queue.cc:237] T 49eb89bf68d942398c68e2bf1d0f6370 P 33be9a5b29214a8d84c2fd7a4936a920 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "0dfc68b63d15462398c76bfbff0e3cf8" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 40439 } attrs { promote: true } }
I20240502 20:45:40.642407 2695 raft_consensus.cc:2932] T 49eb89bf68d942398c68e2bf1d0f6370 P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0dfc68b63d15462398c76bfbff0e3cf8 (127.30.32.71) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "0dfc68b63d15462398c76bfbff0e3cf8" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 40439 } attrs { promote: true } } }
I20240502 20:45:40.645969 750 catalog_manager.cc:5484] T 49eb89bf68d942398c68e2bf1d0f6370 P 33be9a5b29214a8d84c2fd7a4936a920 reported cstate change: config changed from index -1 to 4, NON_VOTER 0dfc68b63d15462398c76bfbff0e3cf8 (127.30.32.71) added. New cstate: current_term: 1 leader_uuid: "33be9a5b29214a8d84c2fd7a4936a920" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dfc68b63d15462398c76bfbff0e3cf8" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 40439 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 20:45:40.648756 829 consensus_peers.cc:485] T 49eb89bf68d942398c68e2bf1d0f6370 P 33be9a5b29214a8d84c2fd7a4936a920 -> Peer 0dfc68b63d15462398c76bfbff0e3cf8 (127.30.32.71:40439): Couldn't send request to peer 0dfc68b63d15462398c76bfbff0e3cf8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 49eb89bf68d942398c68e2bf1d0f6370. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:40.653643 893 consensus_queue.cc:237] T cf7de28bcb20418f99846c121cd88531 P 33be9a5b29214a8d84c2fd7a4936a920 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "cec988e98db64e2f98b030efc16d4108" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 34789 } attrs { promote: true } }
I20240502 20:45:40.654500 2695 raft_consensus.cc:2932] T cf7de28bcb20418f99846c121cd88531 P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cec988e98db64e2f98b030efc16d4108 (127.30.32.69) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "cec988e98db64e2f98b030efc16d4108" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 34789 } attrs { promote: true } } }
I20240502 20:45:40.656809 750 catalog_manager.cc:5484] T cf7de28bcb20418f99846c121cd88531 P 33be9a5b29214a8d84c2fd7a4936a920 reported cstate change: config changed from index -1 to 4, NON_VOTER cec988e98db64e2f98b030efc16d4108 (127.30.32.69) added. New cstate: current_term: 1 leader_uuid: "33be9a5b29214a8d84c2fd7a4936a920" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec988e98db64e2f98b030efc16d4108" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 34789 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 20:45:40.658689 828 consensus_peers.cc:485] T cf7de28bcb20418f99846c121cd88531 P 33be9a5b29214a8d84c2fd7a4936a920 -> Peer cec988e98db64e2f98b030efc16d4108 (127.30.32.69:34789): Couldn't send request to peer cec988e98db64e2f98b030efc16d4108. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cf7de28bcb20418f99846c121cd88531. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:40.664441 1024 consensus_queue.cc:237] T afc220634a164930bdaa8ece818fb2e7 P ed5783865eb74d39bee2a38e9139dea4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } } peers { permanent_uuid: "6d231fa7f1684535af46a9b196111880" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 43291 } attrs { promote: true } }
I20240502 20:45:40.665834 2713 raft_consensus.cc:2932] T afc220634a164930bdaa8ece818fb2e7 P ed5783865eb74d39bee2a38e9139dea4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6d231fa7f1684535af46a9b196111880 (127.30.32.70) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } } peers { permanent_uuid: "6d231fa7f1684535af46a9b196111880" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 43291 } attrs { promote: true } } }
I20240502 20:45:40.668758 750 catalog_manager.cc:5484] T afc220634a164930bdaa8ece818fb2e7 P ed5783865eb74d39bee2a38e9139dea4 reported cstate change: config changed from index -1 to 4, NON_VOTER 6d231fa7f1684535af46a9b196111880 (127.30.32.70) added. New cstate: current_term: 1 leader_uuid: "ed5783865eb74d39bee2a38e9139dea4" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d231fa7f1684535af46a9b196111880" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 43291 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 20:45:40.671324 958 consensus_peers.cc:485] T afc220634a164930bdaa8ece818fb2e7 P ed5783865eb74d39bee2a38e9139dea4 -> Peer 6d231fa7f1684535af46a9b196111880 (127.30.32.70:43291): Couldn't send request to peer 6d231fa7f1684535af46a9b196111880. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: afc220634a164930bdaa8ece818fb2e7. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:40.675529 1024 consensus_queue.cc:237] T a6c9b839b6a346aca42a280aed623913 P ed5783865eb74d39bee2a38e9139dea4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } } peers { permanent_uuid: "f7fd7bf659f84ae184696b891db5221a" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42407 } attrs { promote: true } }
I20240502 20:45:40.676414 2713 raft_consensus.cc:2932] T a6c9b839b6a346aca42a280aed623913 P ed5783865eb74d39bee2a38e9139dea4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f7fd7bf659f84ae184696b891db5221a (127.30.32.67) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } } peers { permanent_uuid: "f7fd7bf659f84ae184696b891db5221a" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42407 } attrs { promote: true } } }
I20240502 20:45:40.680307 750 catalog_manager.cc:5484] T a6c9b839b6a346aca42a280aed623913 P ed5783865eb74d39bee2a38e9139dea4 reported cstate change: config changed from index -1 to 4, NON_VOTER f7fd7bf659f84ae184696b891db5221a (127.30.32.67) added. New cstate: current_term: 1 leader_uuid: "ed5783865eb74d39bee2a38e9139dea4" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7fd7bf659f84ae184696b891db5221a" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42407 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 20:45:40.682612 960 consensus_peers.cc:485] T a6c9b839b6a346aca42a280aed623913 P ed5783865eb74d39bee2a38e9139dea4 -> Peer f7fd7bf659f84ae184696b891db5221a (127.30.32.67:42407): Couldn't send request to peer f7fd7bf659f84ae184696b891db5221a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a6c9b839b6a346aca42a280aed623913. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:40.687889 893 consensus_queue.cc:237] T 21a27f13d4104a8a87aa8b935c6c8dfa P 33be9a5b29214a8d84c2fd7a4936a920 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "2260afde4ccb498189d7b25c00a2c727" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 40723 } attrs { promote: true } }
I20240502 20:45:40.688946 2695 raft_consensus.cc:2932] T 21a27f13d4104a8a87aa8b935c6c8dfa P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2260afde4ccb498189d7b25c00a2c727 (127.30.32.68) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "2260afde4ccb498189d7b25c00a2c727" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 40723 } attrs { promote: true } } }
W20240502 20:45:40.692732 826 consensus_peers.cc:485] T 21a27f13d4104a8a87aa8b935c6c8dfa P 33be9a5b29214a8d84c2fd7a4936a920 -> Peer 2260afde4ccb498189d7b25c00a2c727 (127.30.32.68:40723): Couldn't send request to peer 2260afde4ccb498189d7b25c00a2c727. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 21a27f13d4104a8a87aa8b935c6c8dfa. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:40.694726 750 catalog_manager.cc:5484] T 21a27f13d4104a8a87aa8b935c6c8dfa P 33be9a5b29214a8d84c2fd7a4936a920 reported cstate change: config changed from index -1 to 4, NON_VOTER 2260afde4ccb498189d7b25c00a2c727 (127.30.32.68) added. New cstate: current_term: 1 leader_uuid: "33be9a5b29214a8d84c2fd7a4936a920" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2260afde4ccb498189d7b25c00a2c727" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 40723 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:40.701996 1024 consensus_queue.cc:237] T 7a888b6096c2427f8a7f67b26c3c00b7 P ed5783865eb74d39bee2a38e9139dea4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } } peers { permanent_uuid: "0dfc68b63d15462398c76bfbff0e3cf8" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 40439 } attrs { promote: true } }
I20240502 20:45:40.703081 2714 raft_consensus.cc:2932] T 7a888b6096c2427f8a7f67b26c3c00b7 P ed5783865eb74d39bee2a38e9139dea4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0dfc68b63d15462398c76bfbff0e3cf8 (127.30.32.71) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } } peers { permanent_uuid: "0dfc68b63d15462398c76bfbff0e3cf8" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 40439 } attrs { promote: true } } }
W20240502 20:45:40.706280 960 consensus_peers.cc:485] T 7a888b6096c2427f8a7f67b26c3c00b7 P ed5783865eb74d39bee2a38e9139dea4 -> Peer 0dfc68b63d15462398c76bfbff0e3cf8 (127.30.32.71:40439): Couldn't send request to peer 0dfc68b63d15462398c76bfbff0e3cf8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7a888b6096c2427f8a7f67b26c3c00b7. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:40.741073 750 catalog_manager.cc:5484] T 7a888b6096c2427f8a7f67b26c3c00b7 P ed5783865eb74d39bee2a38e9139dea4 reported cstate change: config changed from index -1 to 4, NON_VOTER 0dfc68b63d15462398c76bfbff0e3cf8 (127.30.32.71) added. New cstate: current_term: 1 leader_uuid: "ed5783865eb74d39bee2a38e9139dea4" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dfc68b63d15462398c76bfbff0e3cf8" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 40439 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:40.748699 893 consensus_queue.cc:237] T 0734f8b2eeb04ad99ff827c0ecf90663 P 33be9a5b29214a8d84c2fd7a4936a920 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "cec988e98db64e2f98b030efc16d4108" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 34789 } attrs { promote: true } }
I20240502 20:45:40.749868 2728 raft_consensus.cc:2932] T 0734f8b2eeb04ad99ff827c0ecf90663 P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cec988e98db64e2f98b030efc16d4108 (127.30.32.69) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "cec988e98db64e2f98b030efc16d4108" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 34789 } attrs { promote: true } } }
W20240502 20:45:40.750370 828 consensus_peers.cc:485] T 0734f8b2eeb04ad99ff827c0ecf90663 P 33be9a5b29214a8d84c2fd7a4936a920 -> Peer cec988e98db64e2f98b030efc16d4108 (127.30.32.69:34789): Couldn't send request to peer cec988e98db64e2f98b030efc16d4108. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0734f8b2eeb04ad99ff827c0ecf90663. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:40.823339 750 catalog_manager.cc:5484] T 0734f8b2eeb04ad99ff827c0ecf90663 P 33be9a5b29214a8d84c2fd7a4936a920 reported cstate change: config changed from index -1 to 4, NON_VOTER cec988e98db64e2f98b030efc16d4108 (127.30.32.69) added. New cstate: current_term: 1 leader_uuid: "33be9a5b29214a8d84c2fd7a4936a920" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec988e98db64e2f98b030efc16d4108" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 34789 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:40.830657 1024 consensus_queue.cc:237] T 74b7ff29e276451985e7cbe72ceff148 P ed5783865eb74d39bee2a38e9139dea4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } } peers { permanent_uuid: "6d231fa7f1684535af46a9b196111880" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 43291 } attrs { promote: true } }
I20240502 20:45:40.831694 2713 raft_consensus.cc:2932] T 74b7ff29e276451985e7cbe72ceff148 P ed5783865eb74d39bee2a38e9139dea4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6d231fa7f1684535af46a9b196111880 (127.30.32.70) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } } peers { permanent_uuid: "6d231fa7f1684535af46a9b196111880" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 43291 } attrs { promote: true } } }
W20240502 20:45:40.832185 958 consensus_peers.cc:485] T 74b7ff29e276451985e7cbe72ceff148 P ed5783865eb74d39bee2a38e9139dea4 -> Peer 6d231fa7f1684535af46a9b196111880 (127.30.32.70:43291): Couldn't send request to peer 6d231fa7f1684535af46a9b196111880. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 74b7ff29e276451985e7cbe72ceff148. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:40.834481 750 catalog_manager.cc:5484] T 74b7ff29e276451985e7cbe72ceff148 P ed5783865eb74d39bee2a38e9139dea4 reported cstate change: config changed from index -1 to 4, NON_VOTER 6d231fa7f1684535af46a9b196111880 (127.30.32.70) added. New cstate: current_term: 1 leader_uuid: "ed5783865eb74d39bee2a38e9139dea4" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d231fa7f1684535af46a9b196111880" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 43291 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:40.842267 2343 heartbeater.cc:503] Master 127.30.32.126:43021 was elected leader, sending a full tablet report...
I20240502 20:45:40.842352 893 consensus_queue.cc:237] T da05217d82a74bd6aeb54330c3d6d370 P 33be9a5b29214a8d84c2fd7a4936a920 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "f7fd7bf659f84ae184696b891db5221a" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42407 } attrs { promote: true } }
I20240502 20:45:40.843497 2728 raft_consensus.cc:2932] T da05217d82a74bd6aeb54330c3d6d370 P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f7fd7bf659f84ae184696b891db5221a (127.30.32.67) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "f7fd7bf659f84ae184696b891db5221a" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42407 } attrs { promote: true } } }
I20240502 20:45:40.845932 750 catalog_manager.cc:5484] T da05217d82a74bd6aeb54330c3d6d370 P 33be9a5b29214a8d84c2fd7a4936a920 reported cstate change: config changed from index -1 to 4, NON_VOTER f7fd7bf659f84ae184696b891db5221a (127.30.32.67) added. New cstate: current_term: 1 leader_uuid: "33be9a5b29214a8d84c2fd7a4936a920" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7fd7bf659f84ae184696b891db5221a" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42407 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 20:45:40.847678 829 consensus_peers.cc:485] T da05217d82a74bd6aeb54330c3d6d370 P 33be9a5b29214a8d84c2fd7a4936a920 -> Peer f7fd7bf659f84ae184696b891db5221a (127.30.32.67:42407): Couldn't send request to peer f7fd7bf659f84ae184696b891db5221a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: da05217d82a74bd6aeb54330c3d6d370. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:41.042486 2474 heartbeater.cc:503] Master 127.30.32.126:43021 was elected leader, sending a full tablet report...
I20240502 20:45:41.096773 2799 ts_tablet_manager.cc:902] T cf7de28bcb20418f99846c121cd88531 P cec988e98db64e2f98b030efc16d4108: Initiating tablet copy from peer 33be9a5b29214a8d84c2fd7a4936a920 (127.30.32.65:45537)
I20240502 20:45:41.097642 2799 tablet_copy_client.cc:308] T cf7de28bcb20418f99846c121cd88531 P cec988e98db64e2f98b030efc16d4108: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.65:45537
I20240502 20:45:41.101131 905 tablet_copy_service.cc:140] P 33be9a5b29214a8d84c2fd7a4936a920: Received BeginTabletCopySession request for tablet cf7de28bcb20418f99846c121cd88531 from peer cec988e98db64e2f98b030efc16d4108 ({username='slave'} at 127.30.32.69:35135)
I20240502 20:45:41.101269 905 tablet_copy_service.cc:161] P 33be9a5b29214a8d84c2fd7a4936a920: Beginning new tablet copy session on tablet cf7de28bcb20418f99846c121cd88531 from peer cec988e98db64e2f98b030efc16d4108 at {username='slave'} at 127.30.32.69:35135: session id = cec988e98db64e2f98b030efc16d4108-cf7de28bcb20418f99846c121cd88531
I20240502 20:45:41.102396 905 tablet_copy_source_session.cc:205] T cf7de28bcb20418f99846c121cd88531 P 33be9a5b29214a8d84c2fd7a4936a920: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:45:41.105577 2799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf7de28bcb20418f99846c121cd88531. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:41.111464 2799 tablet_copy_client.cc:783] T cf7de28bcb20418f99846c121cd88531 P cec988e98db64e2f98b030efc16d4108: tablet copy: Starting download of 0 data blocks...
I20240502 20:45:41.111711 2799 tablet_copy_client.cc:647] T cf7de28bcb20418f99846c121cd88531 P cec988e98db64e2f98b030efc16d4108: tablet copy: Starting download of 1 WAL segments...
I20240502 20:45:41.113456 2799 tablet_copy_client.cc:523] T cf7de28bcb20418f99846c121cd88531 P cec988e98db64e2f98b030efc16d4108: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:45:41.115087 2799 tablet_bootstrap.cc:492] T cf7de28bcb20418f99846c121cd88531 P cec988e98db64e2f98b030efc16d4108: Bootstrap starting.
I20240502 20:45:41.120784 2799 log.cc:824] T cf7de28bcb20418f99846c121cd88531 P cec988e98db64e2f98b030efc16d4108: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:41.139452 2841 ts_tablet_manager.cc:902] T afc220634a164930bdaa8ece818fb2e7 P 6d231fa7f1684535af46a9b196111880: Initiating tablet copy from peer ed5783865eb74d39bee2a38e9139dea4 (127.30.32.66:45113)
I20240502 20:45:41.140309 2841 tablet_copy_client.cc:308] T afc220634a164930bdaa8ece818fb2e7 P 6d231fa7f1684535af46a9b196111880: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.66:45113
I20240502 20:45:41.144765 1044 tablet_copy_service.cc:140] P ed5783865eb74d39bee2a38e9139dea4: Received BeginTabletCopySession request for tablet afc220634a164930bdaa8ece818fb2e7 from peer 6d231fa7f1684535af46a9b196111880 ({username='slave'} at 127.30.32.70:43835)
I20240502 20:45:41.144910 1044 tablet_copy_service.cc:161] P ed5783865eb74d39bee2a38e9139dea4: Beginning new tablet copy session on tablet afc220634a164930bdaa8ece818fb2e7 from peer 6d231fa7f1684535af46a9b196111880 at {username='slave'} at 127.30.32.70:43835: session id = 6d231fa7f1684535af46a9b196111880-afc220634a164930bdaa8ece818fb2e7
I20240502 20:45:41.145315 2799 tablet_bootstrap.cc:492] T cf7de28bcb20418f99846c121cd88531 P cec988e98db64e2f98b030efc16d4108: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:45:41.145725 2799 tablet_bootstrap.cc:492] T cf7de28bcb20418f99846c121cd88531 P cec988e98db64e2f98b030efc16d4108: Bootstrap complete.
I20240502 20:45:41.146061 2799 ts_tablet_manager.cc:1367] T cf7de28bcb20418f99846c121cd88531 P cec988e98db64e2f98b030efc16d4108: Time spent bootstrapping tablet: real 0.031s user 0.027s sys 0.004s
I20240502 20:45:41.146229 1044 tablet_copy_source_session.cc:205] T afc220634a164930bdaa8ece818fb2e7 P ed5783865eb74d39bee2a38e9139dea4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:45:41.148715 2799 raft_consensus.cc:348] T cf7de28bcb20418f99846c121cd88531 P cec988e98db64e2f98b030efc16d4108 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "cec988e98db64e2f98b030efc16d4108" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 34789 } attrs { promote: true } }
I20240502 20:45:41.149011 2799 raft_consensus.cc:729] T cf7de28bcb20418f99846c121cd88531 P cec988e98db64e2f98b030efc16d4108 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cec988e98db64e2f98b030efc16d4108, State: Initialized, Role: LEARNER
I20240502 20:45:41.149406 2799 consensus_queue.cc:260] T cf7de28bcb20418f99846c121cd88531 P cec988e98db64e2f98b030efc16d4108 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "cec988e98db64e2f98b030efc16d4108" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 34789 } attrs { promote: true } }
I20240502 20:45:41.150125 2841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afc220634a164930bdaa8ece818fb2e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:41.150310 2799 ts_tablet_manager.cc:1398] T cf7de28bcb20418f99846c121cd88531 P cec988e98db64e2f98b030efc16d4108: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240502 20:45:41.151108 905 tablet_copy_service.cc:342] P 33be9a5b29214a8d84c2fd7a4936a920: Request end of tablet copy session cec988e98db64e2f98b030efc16d4108-cf7de28bcb20418f99846c121cd88531 received from {username='slave'} at 127.30.32.69:35135
I20240502 20:45:41.151196 905 tablet_copy_service.cc:433] P 33be9a5b29214a8d84c2fd7a4936a920: ending tablet copy session cec988e98db64e2f98b030efc16d4108-cf7de28bcb20418f99846c121cd88531 on tablet cf7de28bcb20418f99846c121cd88531 with peer cec988e98db64e2f98b030efc16d4108
I20240502 20:45:41.156190 2841 tablet_copy_client.cc:783] T afc220634a164930bdaa8ece818fb2e7 P 6d231fa7f1684535af46a9b196111880: tablet copy: Starting download of 0 data blocks...
I20240502 20:45:41.156388 2841 tablet_copy_client.cc:647] T afc220634a164930bdaa8ece818fb2e7 P 6d231fa7f1684535af46a9b196111880: tablet copy: Starting download of 1 WAL segments...
I20240502 20:45:41.157793 2858 ts_tablet_manager.cc:902] T 49eb89bf68d942398c68e2bf1d0f6370 P 0dfc68b63d15462398c76bfbff0e3cf8: Initiating tablet copy from peer 33be9a5b29214a8d84c2fd7a4936a920 (127.30.32.65:45537)
I20240502 20:45:41.158077 2841 tablet_copy_client.cc:523] T afc220634a164930bdaa8ece818fb2e7 P 6d231fa7f1684535af46a9b196111880: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:45:41.158739 2858 tablet_copy_client.cc:308] T 49eb89bf68d942398c68e2bf1d0f6370 P 0dfc68b63d15462398c76bfbff0e3cf8: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.65:45537
I20240502 20:45:41.159698 2841 tablet_bootstrap.cc:492] T afc220634a164930bdaa8ece818fb2e7 P 6d231fa7f1684535af46a9b196111880: Bootstrap starting.
I20240502 20:45:41.162472 905 tablet_copy_service.cc:140] P 33be9a5b29214a8d84c2fd7a4936a920: Received BeginTabletCopySession request for tablet 49eb89bf68d942398c68e2bf1d0f6370 from peer 0dfc68b63d15462398c76bfbff0e3cf8 ({username='slave'} at 127.30.32.71:58745)
I20240502 20:45:41.162572 905 tablet_copy_service.cc:161] P 33be9a5b29214a8d84c2fd7a4936a920: Beginning new tablet copy session on tablet 49eb89bf68d942398c68e2bf1d0f6370 from peer 0dfc68b63d15462398c76bfbff0e3cf8 at {username='slave'} at 127.30.32.71:58745: session id = 0dfc68b63d15462398c76bfbff0e3cf8-49eb89bf68d942398c68e2bf1d0f6370
I20240502 20:45:41.163290 905 tablet_copy_source_session.cc:205] T 49eb89bf68d942398c68e2bf1d0f6370 P 33be9a5b29214a8d84c2fd7a4936a920: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:45:41.165001 2841 log.cc:824] T afc220634a164930bdaa8ece818fb2e7 P 6d231fa7f1684535af46a9b196111880: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:41.165910 2858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49eb89bf68d942398c68e2bf1d0f6370. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:41.169325 2799 ts_tablet_manager.cc:902] T 0734f8b2eeb04ad99ff827c0ecf90663 P cec988e98db64e2f98b030efc16d4108: Initiating tablet copy from peer 33be9a5b29214a8d84c2fd7a4936a920 (127.30.32.65:45537)
I20240502 20:45:41.169780 2799 tablet_copy_client.cc:308] T 0734f8b2eeb04ad99ff827c0ecf90663 P cec988e98db64e2f98b030efc16d4108: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.65:45537
I20240502 20:45:41.170290 905 tablet_copy_service.cc:140] P 33be9a5b29214a8d84c2fd7a4936a920: Received BeginTabletCopySession request for tablet 0734f8b2eeb04ad99ff827c0ecf90663 from peer cec988e98db64e2f98b030efc16d4108 ({username='slave'} at 127.30.32.69:35135)
I20240502 20:45:41.170384 905 tablet_copy_service.cc:161] P 33be9a5b29214a8d84c2fd7a4936a920: Beginning new tablet copy session on tablet 0734f8b2eeb04ad99ff827c0ecf90663 from peer cec988e98db64e2f98b030efc16d4108 at {username='slave'} at 127.30.32.69:35135: session id = cec988e98db64e2f98b030efc16d4108-0734f8b2eeb04ad99ff827c0ecf90663
I20240502 20:45:41.171248 905 tablet_copy_source_session.cc:205] T 0734f8b2eeb04ad99ff827c0ecf90663 P 33be9a5b29214a8d84c2fd7a4936a920: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:45:41.171399 2858 tablet_copy_client.cc:783] T 49eb89bf68d942398c68e2bf1d0f6370 P 0dfc68b63d15462398c76bfbff0e3cf8: tablet copy: Starting download of 0 data blocks...
I20240502 20:45:41.171558 2858 tablet_copy_client.cc:647] T 49eb89bf68d942398c68e2bf1d0f6370 P 0dfc68b63d15462398c76bfbff0e3cf8: tablet copy: Starting download of 1 WAL segments...
I20240502 20:45:41.173820 2799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0734f8b2eeb04ad99ff827c0ecf90663. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:41.174216 2858 tablet_copy_client.cc:523] T 49eb89bf68d942398c68e2bf1d0f6370 P 0dfc68b63d15462398c76bfbff0e3cf8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:45:41.175726 2799 tablet_copy_client.cc:783] T 0734f8b2eeb04ad99ff827c0ecf90663 P cec988e98db64e2f98b030efc16d4108: tablet copy: Starting download of 0 data blocks...
I20240502 20:45:41.175863 2799 tablet_copy_client.cc:647] T 0734f8b2eeb04ad99ff827c0ecf90663 P cec988e98db64e2f98b030efc16d4108: tablet copy: Starting download of 1 WAL segments...
I20240502 20:45:41.175925 2858 tablet_bootstrap.cc:492] T 49eb89bf68d942398c68e2bf1d0f6370 P 0dfc68b63d15462398c76bfbff0e3cf8: Bootstrap starting.
I20240502 20:45:41.179237 2799 tablet_copy_client.cc:523] T 0734f8b2eeb04ad99ff827c0ecf90663 P cec988e98db64e2f98b030efc16d4108: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:45:41.180328 2799 tablet_bootstrap.cc:492] T 0734f8b2eeb04ad99ff827c0ecf90663 P cec988e98db64e2f98b030efc16d4108: Bootstrap starting.
I20240502 20:45:41.182559 2858 log.cc:824] T 49eb89bf68d942398c68e2bf1d0f6370 P 0dfc68b63d15462398c76bfbff0e3cf8: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:41.191716 2841 tablet_bootstrap.cc:492] T afc220634a164930bdaa8ece818fb2e7 P 6d231fa7f1684535af46a9b196111880: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:45:41.192157 2841 tablet_bootstrap.cc:492] T afc220634a164930bdaa8ece818fb2e7 P 6d231fa7f1684535af46a9b196111880: Bootstrap complete.
I20240502 20:45:41.192509 2841 ts_tablet_manager.cc:1367] T afc220634a164930bdaa8ece818fb2e7 P 6d231fa7f1684535af46a9b196111880: Time spent bootstrapping tablet: real 0.033s user 0.025s sys 0.009s
I20240502 20:45:41.194962 2841 raft_consensus.cc:348] T afc220634a164930bdaa8ece818fb2e7 P 6d231fa7f1684535af46a9b196111880 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } } peers { permanent_uuid: "6d231fa7f1684535af46a9b196111880" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 43291 } attrs { promote: true } }
I20240502 20:45:41.195266 2841 raft_consensus.cc:729] T afc220634a164930bdaa8ece818fb2e7 P 6d231fa7f1684535af46a9b196111880 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6d231fa7f1684535af46a9b196111880, State: Initialized, Role: LEARNER
I20240502 20:45:41.195632 2841 consensus_queue.cc:260] T afc220634a164930bdaa8ece818fb2e7 P 6d231fa7f1684535af46a9b196111880 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } } peers { permanent_uuid: "6d231fa7f1684535af46a9b196111880" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 43291 } attrs { promote: true } }
I20240502 20:45:41.196544 2841 ts_tablet_manager.cc:1398] T afc220634a164930bdaa8ece818fb2e7 P 6d231fa7f1684535af46a9b196111880: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 20:45:41.197464 1044 tablet_copy_service.cc:342] P ed5783865eb74d39bee2a38e9139dea4: Request end of tablet copy session 6d231fa7f1684535af46a9b196111880-afc220634a164930bdaa8ece818fb2e7 received from {username='slave'} at 127.30.32.70:43835
I20240502 20:45:41.197553 1044 tablet_copy_service.cc:433] P ed5783865eb74d39bee2a38e9139dea4: ending tablet copy session 6d231fa7f1684535af46a9b196111880-afc220634a164930bdaa8ece818fb2e7 on tablet afc220634a164930bdaa8ece818fb2e7 with peer 6d231fa7f1684535af46a9b196111880
I20240502 20:45:41.204792 2799 tablet_bootstrap.cc:492] T 0734f8b2eeb04ad99ff827c0ecf90663 P cec988e98db64e2f98b030efc16d4108: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:45:41.205085 2799 tablet_bootstrap.cc:492] T 0734f8b2eeb04ad99ff827c0ecf90663 P cec988e98db64e2f98b030efc16d4108: Bootstrap complete.
I20240502 20:45:41.205276 2799 ts_tablet_manager.cc:1367] T 0734f8b2eeb04ad99ff827c0ecf90663 P cec988e98db64e2f98b030efc16d4108: Time spent bootstrapping tablet: real 0.025s user 0.021s sys 0.003s
I20240502 20:45:41.205657 2858 tablet_bootstrap.cc:492] T 49eb89bf68d942398c68e2bf1d0f6370 P 0dfc68b63d15462398c76bfbff0e3cf8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:45:41.205792 2799 raft_consensus.cc:348] T 0734f8b2eeb04ad99ff827c0ecf90663 P cec988e98db64e2f98b030efc16d4108 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "cec988e98db64e2f98b030efc16d4108" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 34789 } attrs { promote: true } }
I20240502 20:45:41.205893 2799 raft_consensus.cc:729] T 0734f8b2eeb04ad99ff827c0ecf90663 P cec988e98db64e2f98b030efc16d4108 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cec988e98db64e2f98b030efc16d4108, State: Initialized, Role: LEARNER
I20240502 20:45:41.206046 2858 tablet_bootstrap.cc:492] T 49eb89bf68d942398c68e2bf1d0f6370 P 0dfc68b63d15462398c76bfbff0e3cf8: Bootstrap complete.
I20240502 20:45:41.206020 2799 consensus_queue.cc:260] T 0734f8b2eeb04ad99ff827c0ecf90663 P cec988e98db64e2f98b030efc16d4108 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "cec988e98db64e2f98b030efc16d4108" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 34789 } attrs { promote: true } }
I20240502 20:45:41.206305 2799 ts_tablet_manager.cc:1398] T 0734f8b2eeb04ad99ff827c0ecf90663 P cec988e98db64e2f98b030efc16d4108: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:41.206372 2858 ts_tablet_manager.cc:1367] T 49eb89bf68d942398c68e2bf1d0f6370 P 0dfc68b63d15462398c76bfbff0e3cf8: Time spent bootstrapping tablet: real 0.031s user 0.028s sys 0.000s
I20240502 20:45:41.206902 905 tablet_copy_service.cc:342] P 33be9a5b29214a8d84c2fd7a4936a920: Request end of tablet copy session cec988e98db64e2f98b030efc16d4108-0734f8b2eeb04ad99ff827c0ecf90663 received from {username='slave'} at 127.30.32.69:35135
I20240502 20:45:41.206992 905 tablet_copy_service.cc:433] P 33be9a5b29214a8d84c2fd7a4936a920: ending tablet copy session cec988e98db64e2f98b030efc16d4108-0734f8b2eeb04ad99ff827c0ecf90663 on tablet 0734f8b2eeb04ad99ff827c0ecf90663 with peer cec988e98db64e2f98b030efc16d4108
I20240502 20:45:41.208597 2858 raft_consensus.cc:348] T 49eb89bf68d942398c68e2bf1d0f6370 P 0dfc68b63d15462398c76bfbff0e3cf8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "0dfc68b63d15462398c76bfbff0e3cf8" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 40439 } attrs { promote: true } }
I20240502 20:45:41.208797 2858 raft_consensus.cc:729] T 49eb89bf68d942398c68e2bf1d0f6370 P 0dfc68b63d15462398c76bfbff0e3cf8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0dfc68b63d15462398c76bfbff0e3cf8, State: Initialized, Role: LEARNER
I20240502 20:45:41.209146 2858 consensus_queue.cc:260] T 49eb89bf68d942398c68e2bf1d0f6370 P 0dfc68b63d15462398c76bfbff0e3cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "0dfc68b63d15462398c76bfbff0e3cf8" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 40439 } attrs { promote: true } }
I20240502 20:45:41.209924 2605 heartbeater.cc:503] Master 127.30.32.126:43021 was elected leader, sending a full tablet report...
I20240502 20:45:41.209954 2873 ts_tablet_manager.cc:902] T 7a888b6096c2427f8a7f67b26c3c00b7 P 0dfc68b63d15462398c76bfbff0e3cf8: Initiating tablet copy from peer ed5783865eb74d39bee2a38e9139dea4 (127.30.32.66:45113)
I20240502 20:45:41.209965 2858 ts_tablet_manager.cc:1398] T 49eb89bf68d942398c68e2bf1d0f6370 P 0dfc68b63d15462398c76bfbff0e3cf8: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240502 20:45:41.210598 2873 tablet_copy_client.cc:308] T 7a888b6096c2427f8a7f67b26c3c00b7 P 0dfc68b63d15462398c76bfbff0e3cf8: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.66:45113
I20240502 20:45:41.210687 905 tablet_copy_service.cc:342] P 33be9a5b29214a8d84c2fd7a4936a920: Request end of tablet copy session 0dfc68b63d15462398c76bfbff0e3cf8-49eb89bf68d942398c68e2bf1d0f6370 received from {username='slave'} at 127.30.32.71:58745
I20240502 20:45:41.210778 905 tablet_copy_service.cc:433] P 33be9a5b29214a8d84c2fd7a4936a920: ending tablet copy session 0dfc68b63d15462398c76bfbff0e3cf8-49eb89bf68d942398c68e2bf1d0f6370 on tablet 49eb89bf68d942398c68e2bf1d0f6370 with peer 0dfc68b63d15462398c76bfbff0e3cf8
I20240502 20:45:41.214174 1044 tablet_copy_service.cc:140] P ed5783865eb74d39bee2a38e9139dea4: Received BeginTabletCopySession request for tablet 7a888b6096c2427f8a7f67b26c3c00b7 from peer 0dfc68b63d15462398c76bfbff0e3cf8 ({username='slave'} at 127.30.32.71:42549)
I20240502 20:45:41.214270 1044 tablet_copy_service.cc:161] P ed5783865eb74d39bee2a38e9139dea4: Beginning new tablet copy session on tablet 7a888b6096c2427f8a7f67b26c3c00b7 from peer 0dfc68b63d15462398c76bfbff0e3cf8 at {username='slave'} at 127.30.32.71:42549: session id = 0dfc68b63d15462398c76bfbff0e3cf8-7a888b6096c2427f8a7f67b26c3c00b7
I20240502 20:45:41.214946 1044 tablet_copy_source_session.cc:205] T 7a888b6096c2427f8a7f67b26c3c00b7 P ed5783865eb74d39bee2a38e9139dea4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:45:41.215652 2873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a888b6096c2427f8a7f67b26c3c00b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:41.217648 2873 tablet_copy_client.cc:783] T 7a888b6096c2427f8a7f67b26c3c00b7 P 0dfc68b63d15462398c76bfbff0e3cf8: tablet copy: Starting download of 0 data blocks...
I20240502 20:45:41.217800 2873 tablet_copy_client.cc:647] T 7a888b6096c2427f8a7f67b26c3c00b7 P 0dfc68b63d15462398c76bfbff0e3cf8: tablet copy: Starting download of 1 WAL segments...
I20240502 20:45:41.218883 2873 tablet_copy_client.cc:523] T 7a888b6096c2427f8a7f67b26c3c00b7 P 0dfc68b63d15462398c76bfbff0e3cf8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:45:41.219897 2873 tablet_bootstrap.cc:492] T 7a888b6096c2427f8a7f67b26c3c00b7 P 0dfc68b63d15462398c76bfbff0e3cf8: Bootstrap starting.
I20240502 20:45:41.221601 2878 ts_tablet_manager.cc:902] T a6c9b839b6a346aca42a280aed623913 P f7fd7bf659f84ae184696b891db5221a: Initiating tablet copy from peer ed5783865eb74d39bee2a38e9139dea4 (127.30.32.66:45113)
I20240502 20:45:41.222422 2878 tablet_copy_client.cc:308] T a6c9b839b6a346aca42a280aed623913 P f7fd7bf659f84ae184696b891db5221a: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.66:45113
I20240502 20:45:41.228437 1044 tablet_copy_service.cc:140] P ed5783865eb74d39bee2a38e9139dea4: Received BeginTabletCopySession request for tablet a6c9b839b6a346aca42a280aed623913 from peer f7fd7bf659f84ae184696b891db5221a ({username='slave'} at 127.30.32.67:46445)
I20240502 20:45:41.228529 1044 tablet_copy_service.cc:161] P ed5783865eb74d39bee2a38e9139dea4: Beginning new tablet copy session on tablet a6c9b839b6a346aca42a280aed623913 from peer f7fd7bf659f84ae184696b891db5221a at {username='slave'} at 127.30.32.67:46445: session id = f7fd7bf659f84ae184696b891db5221a-a6c9b839b6a346aca42a280aed623913
I20240502 20:45:41.229297 1044 tablet_copy_source_session.cc:205] T a6c9b839b6a346aca42a280aed623913 P ed5783865eb74d39bee2a38e9139dea4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:45:41.232687 2878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6c9b839b6a346aca42a280aed623913. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:41.238842 2878 tablet_copy_client.cc:783] T a6c9b839b6a346aca42a280aed623913 P f7fd7bf659f84ae184696b891db5221a: tablet copy: Starting download of 0 data blocks...
I20240502 20:45:41.239097 2878 tablet_copy_client.cc:647] T a6c9b839b6a346aca42a280aed623913 P f7fd7bf659f84ae184696b891db5221a: tablet copy: Starting download of 1 WAL segments...
I20240502 20:45:41.239738 2873 tablet_bootstrap.cc:492] T 7a888b6096c2427f8a7f67b26c3c00b7 P 0dfc68b63d15462398c76bfbff0e3cf8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:45:41.240043 2873 tablet_bootstrap.cc:492] T 7a888b6096c2427f8a7f67b26c3c00b7 P 0dfc68b63d15462398c76bfbff0e3cf8: Bootstrap complete.
I20240502 20:45:41.240232 2873 ts_tablet_manager.cc:1367] T 7a888b6096c2427f8a7f67b26c3c00b7 P 0dfc68b63d15462398c76bfbff0e3cf8: Time spent bootstrapping tablet: real 0.020s user 0.014s sys 0.004s
I20240502 20:45:41.240677 2878 tablet_copy_client.cc:523] T a6c9b839b6a346aca42a280aed623913 P f7fd7bf659f84ae184696b891db5221a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:45:41.240780 2873 raft_consensus.cc:348] T 7a888b6096c2427f8a7f67b26c3c00b7 P 0dfc68b63d15462398c76bfbff0e3cf8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } } peers { permanent_uuid: "0dfc68b63d15462398c76bfbff0e3cf8" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 40439 } attrs { promote: true } }
I20240502 20:45:41.240900 2873 raft_consensus.cc:729] T 7a888b6096c2427f8a7f67b26c3c00b7 P 0dfc68b63d15462398c76bfbff0e3cf8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0dfc68b63d15462398c76bfbff0e3cf8, State: Initialized, Role: LEARNER
I20240502 20:45:41.241062 2873 consensus_queue.cc:260] T 7a888b6096c2427f8a7f67b26c3c00b7 P 0dfc68b63d15462398c76bfbff0e3cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } } peers { permanent_uuid: "0dfc68b63d15462398c76bfbff0e3cf8" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 40439 } attrs { promote: true } }
I20240502 20:45:41.241534 2873 ts_tablet_manager.cc:1398] T 7a888b6096c2427f8a7f67b26c3c00b7 P 0dfc68b63d15462398c76bfbff0e3cf8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:41.242408 2878 tablet_bootstrap.cc:492] T a6c9b839b6a346aca42a280aed623913 P f7fd7bf659f84ae184696b891db5221a: Bootstrap starting.
I20240502 20:45:41.243247 1044 tablet_copy_service.cc:342] P ed5783865eb74d39bee2a38e9139dea4: Request end of tablet copy session 0dfc68b63d15462398c76bfbff0e3cf8-7a888b6096c2427f8a7f67b26c3c00b7 received from {username='slave'} at 127.30.32.71:42549
I20240502 20:45:41.243330 1044 tablet_copy_service.cc:433] P ed5783865eb74d39bee2a38e9139dea4: ending tablet copy session 0dfc68b63d15462398c76bfbff0e3cf8-7a888b6096c2427f8a7f67b26c3c00b7 on tablet 7a888b6096c2427f8a7f67b26c3c00b7 with peer 0dfc68b63d15462398c76bfbff0e3cf8
I20240502 20:45:41.243337 2841 ts_tablet_manager.cc:902] T 74b7ff29e276451985e7cbe72ceff148 P 6d231fa7f1684535af46a9b196111880: Initiating tablet copy from peer ed5783865eb74d39bee2a38e9139dea4 (127.30.32.66:45113)
I20240502 20:45:41.243779 2841 tablet_copy_client.cc:308] T 74b7ff29e276451985e7cbe72ceff148 P 6d231fa7f1684535af46a9b196111880: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.66:45113
I20240502 20:45:41.244174 1044 tablet_copy_service.cc:140] P ed5783865eb74d39bee2a38e9139dea4: Received BeginTabletCopySession request for tablet 74b7ff29e276451985e7cbe72ceff148 from peer 6d231fa7f1684535af46a9b196111880 ({username='slave'} at 127.30.32.70:43835)
I20240502 20:45:41.244287 1044 tablet_copy_service.cc:161] P ed5783865eb74d39bee2a38e9139dea4: Beginning new tablet copy session on tablet 74b7ff29e276451985e7cbe72ceff148 from peer 6d231fa7f1684535af46a9b196111880 at {username='slave'} at 127.30.32.70:43835: session id = 6d231fa7f1684535af46a9b196111880-74b7ff29e276451985e7cbe72ceff148
I20240502 20:45:41.246037 1044 tablet_copy_source_session.cc:205] T 74b7ff29e276451985e7cbe72ceff148 P ed5783865eb74d39bee2a38e9139dea4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:45:41.246860 2841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74b7ff29e276451985e7cbe72ceff148. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:41.248498 2878 log.cc:824] T a6c9b839b6a346aca42a280aed623913 P f7fd7bf659f84ae184696b891db5221a: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:41.248565 2841 tablet_copy_client.cc:783] T 74b7ff29e276451985e7cbe72ceff148 P 6d231fa7f1684535af46a9b196111880: tablet copy: Starting download of 0 data blocks...
I20240502 20:45:41.248701 2841 tablet_copy_client.cc:647] T 74b7ff29e276451985e7cbe72ceff148 P 6d231fa7f1684535af46a9b196111880: tablet copy: Starting download of 1 WAL segments...
I20240502 20:45:41.249810 2841 tablet_copy_client.cc:523] T 74b7ff29e276451985e7cbe72ceff148 P 6d231fa7f1684535af46a9b196111880: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:45:41.250826 2841 tablet_bootstrap.cc:492] T 74b7ff29e276451985e7cbe72ceff148 P 6d231fa7f1684535af46a9b196111880: Bootstrap starting.
I20240502 20:45:41.267401 2841 tablet_bootstrap.cc:492] T 74b7ff29e276451985e7cbe72ceff148 P 6d231fa7f1684535af46a9b196111880: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:45:41.267679 2841 tablet_bootstrap.cc:492] T 74b7ff29e276451985e7cbe72ceff148 P 6d231fa7f1684535af46a9b196111880: Bootstrap complete.
I20240502 20:45:41.267849 2841 ts_tablet_manager.cc:1367] T 74b7ff29e276451985e7cbe72ceff148 P 6d231fa7f1684535af46a9b196111880: Time spent bootstrapping tablet: real 0.017s user 0.015s sys 0.000s
I20240502 20:45:41.268477 2878 tablet_bootstrap.cc:492] T a6c9b839b6a346aca42a280aed623913 P f7fd7bf659f84ae184696b891db5221a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:45:41.268436 2841 raft_consensus.cc:348] T 74b7ff29e276451985e7cbe72ceff148 P 6d231fa7f1684535af46a9b196111880 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } } peers { permanent_uuid: "6d231fa7f1684535af46a9b196111880" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 43291 } attrs { promote: true } }
I20240502 20:45:41.268543 2841 raft_consensus.cc:729] T 74b7ff29e276451985e7cbe72ceff148 P 6d231fa7f1684535af46a9b196111880 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6d231fa7f1684535af46a9b196111880, State: Initialized, Role: LEARNER
I20240502 20:45:41.268646 2841 consensus_queue.cc:260] T 74b7ff29e276451985e7cbe72ceff148 P 6d231fa7f1684535af46a9b196111880 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } } peers { permanent_uuid: "6d231fa7f1684535af46a9b196111880" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 43291 } attrs { promote: true } }
I20240502 20:45:41.268790 2878 tablet_bootstrap.cc:492] T a6c9b839b6a346aca42a280aed623913 P f7fd7bf659f84ae184696b891db5221a: Bootstrap complete.
I20240502 20:45:41.268937 2841 ts_tablet_manager.cc:1398] T 74b7ff29e276451985e7cbe72ceff148 P 6d231fa7f1684535af46a9b196111880: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:45:41.269130 2878 ts_tablet_manager.cc:1367] T a6c9b839b6a346aca42a280aed623913 P f7fd7bf659f84ae184696b891db5221a: Time spent bootstrapping tablet: real 0.027s user 0.027s sys 0.000s
I20240502 20:45:41.270493 1044 tablet_copy_service.cc:342] P ed5783865eb74d39bee2a38e9139dea4: Request end of tablet copy session 6d231fa7f1684535af46a9b196111880-74b7ff29e276451985e7cbe72ceff148 received from {username='slave'} at 127.30.32.70:43835
I20240502 20:45:41.270582 1044 tablet_copy_service.cc:433] P ed5783865eb74d39bee2a38e9139dea4: ending tablet copy session 6d231fa7f1684535af46a9b196111880-74b7ff29e276451985e7cbe72ceff148 on tablet 74b7ff29e276451985e7cbe72ceff148 with peer 6d231fa7f1684535af46a9b196111880
I20240502 20:45:41.271793 2878 raft_consensus.cc:348] T a6c9b839b6a346aca42a280aed623913 P f7fd7bf659f84ae184696b891db5221a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } } peers { permanent_uuid: "f7fd7bf659f84ae184696b891db5221a" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42407 } attrs { promote: true } }
I20240502 20:45:41.271997 2878 raft_consensus.cc:729] T a6c9b839b6a346aca42a280aed623913 P f7fd7bf659f84ae184696b891db5221a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f7fd7bf659f84ae184696b891db5221a, State: Initialized, Role: LEARNER
I20240502 20:45:41.272315 2878 consensus_queue.cc:260] T a6c9b839b6a346aca42a280aed623913 P f7fd7bf659f84ae184696b891db5221a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } } peers { permanent_uuid: "f7fd7bf659f84ae184696b891db5221a" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42407 } attrs { promote: true } }
I20240502 20:45:41.273101 2878 ts_tablet_manager.cc:1398] T a6c9b839b6a346aca42a280aed623913 P f7fd7bf659f84ae184696b891db5221a: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240502 20:45:41.273713 1044 tablet_copy_service.cc:342] P ed5783865eb74d39bee2a38e9139dea4: Request end of tablet copy session f7fd7bf659f84ae184696b891db5221a-a6c9b839b6a346aca42a280aed623913 received from {username='slave'} at 127.30.32.67:46445
I20240502 20:45:41.273821 1044 tablet_copy_service.cc:433] P ed5783865eb74d39bee2a38e9139dea4: ending tablet copy session f7fd7bf659f84ae184696b891db5221a-a6c9b839b6a346aca42a280aed623913 on tablet a6c9b839b6a346aca42a280aed623913 with peer f7fd7bf659f84ae184696b891db5221a
I20240502 20:45:41.275907 2892 ts_tablet_manager.cc:902] T 21a27f13d4104a8a87aa8b935c6c8dfa P 2260afde4ccb498189d7b25c00a2c727: Initiating tablet copy from peer 33be9a5b29214a8d84c2fd7a4936a920 (127.30.32.65:45537)
I20240502 20:45:41.276573 2892 tablet_copy_client.cc:308] T 21a27f13d4104a8a87aa8b935c6c8dfa P 2260afde4ccb498189d7b25c00a2c727: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.65:45537
I20240502 20:45:41.280043 905 tablet_copy_service.cc:140] P 33be9a5b29214a8d84c2fd7a4936a920: Received BeginTabletCopySession request for tablet 21a27f13d4104a8a87aa8b935c6c8dfa from peer 2260afde4ccb498189d7b25c00a2c727 ({username='slave'} at 127.30.32.68:43243)
I20240502 20:45:41.280162 905 tablet_copy_service.cc:161] P 33be9a5b29214a8d84c2fd7a4936a920: Beginning new tablet copy session on tablet 21a27f13d4104a8a87aa8b935c6c8dfa from peer 2260afde4ccb498189d7b25c00a2c727 at {username='slave'} at 127.30.32.68:43243: session id = 2260afde4ccb498189d7b25c00a2c727-21a27f13d4104a8a87aa8b935c6c8dfa
I20240502 20:45:41.280936 905 tablet_copy_source_session.cc:205] T 21a27f13d4104a8a87aa8b935c6c8dfa P 33be9a5b29214a8d84c2fd7a4936a920: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:45:41.282426 2892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21a27f13d4104a8a87aa8b935c6c8dfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:41.284950 2892 tablet_copy_client.cc:783] T 21a27f13d4104a8a87aa8b935c6c8dfa P 2260afde4ccb498189d7b25c00a2c727: tablet copy: Starting download of 0 data blocks...
I20240502 20:45:41.285172 2892 tablet_copy_client.cc:647] T 21a27f13d4104a8a87aa8b935c6c8dfa P 2260afde4ccb498189d7b25c00a2c727: tablet copy: Starting download of 1 WAL segments...
I20240502 20:45:41.286597 2892 tablet_copy_client.cc:523] T 21a27f13d4104a8a87aa8b935c6c8dfa P 2260afde4ccb498189d7b25c00a2c727: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:45:41.287711 2892 tablet_bootstrap.cc:492] T 21a27f13d4104a8a87aa8b935c6c8dfa P 2260afde4ccb498189d7b25c00a2c727: Bootstrap starting.
I20240502 20:45:41.288939 2878 ts_tablet_manager.cc:902] T da05217d82a74bd6aeb54330c3d6d370 P f7fd7bf659f84ae184696b891db5221a: Initiating tablet copy from peer 33be9a5b29214a8d84c2fd7a4936a920 (127.30.32.65:45537)
I20240502 20:45:41.289433 2878 tablet_copy_client.cc:308] T da05217d82a74bd6aeb54330c3d6d370 P f7fd7bf659f84ae184696b891db5221a: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.65:45537
I20240502 20:45:41.292814 905 tablet_copy_service.cc:140] P 33be9a5b29214a8d84c2fd7a4936a920: Received BeginTabletCopySession request for tablet da05217d82a74bd6aeb54330c3d6d370 from peer f7fd7bf659f84ae184696b891db5221a ({username='slave'} at 127.30.32.67:47901)
I20240502 20:45:41.292906 905 tablet_copy_service.cc:161] P 33be9a5b29214a8d84c2fd7a4936a920: Beginning new tablet copy session on tablet da05217d82a74bd6aeb54330c3d6d370 from peer f7fd7bf659f84ae184696b891db5221a at {username='slave'} at 127.30.32.67:47901: session id = f7fd7bf659f84ae184696b891db5221a-da05217d82a74bd6aeb54330c3d6d370
I20240502 20:45:41.293608 905 tablet_copy_source_session.cc:205] T da05217d82a74bd6aeb54330c3d6d370 P 33be9a5b29214a8d84c2fd7a4936a920: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:45:41.294396 2878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da05217d82a74bd6aeb54330c3d6d370. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:41.295976 2878 tablet_copy_client.cc:783] T da05217d82a74bd6aeb54330c3d6d370 P f7fd7bf659f84ae184696b891db5221a: tablet copy: Starting download of 0 data blocks...
I20240502 20:45:41.296128 2878 tablet_copy_client.cc:647] T da05217d82a74bd6aeb54330c3d6d370 P f7fd7bf659f84ae184696b891db5221a: tablet copy: Starting download of 1 WAL segments...
I20240502 20:45:41.297243 2878 tablet_copy_client.cc:523] T da05217d82a74bd6aeb54330c3d6d370 P f7fd7bf659f84ae184696b891db5221a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:45:41.298159 2878 tablet_bootstrap.cc:492] T da05217d82a74bd6aeb54330c3d6d370 P f7fd7bf659f84ae184696b891db5221a: Bootstrap starting.
I20240502 20:45:41.309372 2892 tablet_bootstrap.cc:492] T 21a27f13d4104a8a87aa8b935c6c8dfa P 2260afde4ccb498189d7b25c00a2c727: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:45:41.309666 2892 tablet_bootstrap.cc:492] T 21a27f13d4104a8a87aa8b935c6c8dfa P 2260afde4ccb498189d7b25c00a2c727: Bootstrap complete.
I20240502 20:45:41.309839 2892 ts_tablet_manager.cc:1367] T 21a27f13d4104a8a87aa8b935c6c8dfa P 2260afde4ccb498189d7b25c00a2c727: Time spent bootstrapping tablet: real 0.022s user 0.022s sys 0.000s
I20240502 20:45:41.310321 2892 raft_consensus.cc:348] T 21a27f13d4104a8a87aa8b935c6c8dfa P 2260afde4ccb498189d7b25c00a2c727 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "2260afde4ccb498189d7b25c00a2c727" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 40723 } attrs { promote: true } }
I20240502 20:45:41.310426 2892 raft_consensus.cc:729] T 21a27f13d4104a8a87aa8b935c6c8dfa P 2260afde4ccb498189d7b25c00a2c727 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2260afde4ccb498189d7b25c00a2c727, State: Initialized, Role: LEARNER
I20240502 20:45:41.310520 2892 consensus_queue.cc:260] T 21a27f13d4104a8a87aa8b935c6c8dfa P 2260afde4ccb498189d7b25c00a2c727 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "2260afde4ccb498189d7b25c00a2c727" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 40723 } attrs { promote: true } }
I20240502 20:45:41.310911 2892 ts_tablet_manager.cc:1398] T 21a27f13d4104a8a87aa8b935c6c8dfa P 2260afde4ccb498189d7b25c00a2c727: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 20:45:41.311522 905 tablet_copy_service.cc:342] P 33be9a5b29214a8d84c2fd7a4936a920: Request end of tablet copy session 2260afde4ccb498189d7b25c00a2c727-21a27f13d4104a8a87aa8b935c6c8dfa received from {username='slave'} at 127.30.32.68:43243
I20240502 20:45:41.311616 905 tablet_copy_service.cc:433] P 33be9a5b29214a8d84c2fd7a4936a920: ending tablet copy session 2260afde4ccb498189d7b25c00a2c727-21a27f13d4104a8a87aa8b935c6c8dfa on tablet 21a27f13d4104a8a87aa8b935c6c8dfa with peer 2260afde4ccb498189d7b25c00a2c727
I20240502 20:45:41.315881 2878 tablet_bootstrap.cc:492] T da05217d82a74bd6aeb54330c3d6d370 P f7fd7bf659f84ae184696b891db5221a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:45:41.316131 2878 tablet_bootstrap.cc:492] T da05217d82a74bd6aeb54330c3d6d370 P f7fd7bf659f84ae184696b891db5221a: Bootstrap complete.
I20240502 20:45:41.316298 2878 ts_tablet_manager.cc:1367] T da05217d82a74bd6aeb54330c3d6d370 P f7fd7bf659f84ae184696b891db5221a: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.001s
I20240502 20:45:41.316784 2878 raft_consensus.cc:348] T da05217d82a74bd6aeb54330c3d6d370 P f7fd7bf659f84ae184696b891db5221a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "f7fd7bf659f84ae184696b891db5221a" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42407 } attrs { promote: true } }
I20240502 20:45:41.316885 2878 raft_consensus.cc:729] T da05217d82a74bd6aeb54330c3d6d370 P f7fd7bf659f84ae184696b891db5221a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f7fd7bf659f84ae184696b891db5221a, State: Initialized, Role: LEARNER
I20240502 20:45:41.317031 2878 consensus_queue.cc:260] T da05217d82a74bd6aeb54330c3d6d370 P f7fd7bf659f84ae184696b891db5221a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "f7fd7bf659f84ae184696b891db5221a" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42407 } attrs { promote: true } }
I20240502 20:45:41.317337 2878 ts_tablet_manager.cc:1398] T da05217d82a74bd6aeb54330c3d6d370 P f7fd7bf659f84ae184696b891db5221a: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240502 20:45:41.317878 905 tablet_copy_service.cc:342] P 33be9a5b29214a8d84c2fd7a4936a920: Request end of tablet copy session f7fd7bf659f84ae184696b891db5221a-da05217d82a74bd6aeb54330c3d6d370 received from {username='slave'} at 127.30.32.67:47901
I20240502 20:45:41.317957 905 tablet_copy_service.cc:433] P 33be9a5b29214a8d84c2fd7a4936a920: ending tablet copy session f7fd7bf659f84ae184696b891db5221a-da05217d82a74bd6aeb54330c3d6d370 on tablet da05217d82a74bd6aeb54330c3d6d370 with peer f7fd7bf659f84ae184696b891db5221a
I20240502 20:45:41.550698 2297 raft_consensus.cc:1203] T 0734f8b2eeb04ad99ff827c0ecf90663 P cec988e98db64e2f98b030efc16d4108 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:45:41.564100 2559 raft_consensus.cc:1203] T 49eb89bf68d942398c68e2bf1d0f6370 P 0dfc68b63d15462398c76bfbff0e3cf8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:45:41.570430 2428 raft_consensus.cc:1203] T afc220634a164930bdaa8ece818fb2e7 P 6d231fa7f1684535af46a9b196111880 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:45:41.643157 2297 raft_consensus.cc:1203] T cf7de28bcb20418f99846c121cd88531 P cec988e98db64e2f98b030efc16d4108 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:45:41.719823 2157 raft_consensus.cc:1203] T 21a27f13d4104a8a87aa8b935c6c8dfa P 2260afde4ccb498189d7b25c00a2c727 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:45:41.776412 2559 raft_consensus.cc:1203] T 7a888b6096c2427f8a7f67b26c3c00b7 P 0dfc68b63d15462398c76bfbff0e3cf8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:45:41.792948 2025 raft_consensus.cc:1203] T a6c9b839b6a346aca42a280aed623913 P f7fd7bf659f84ae184696b891db5221a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:45:41.818271 2428 raft_consensus.cc:1203] T 74b7ff29e276451985e7cbe72ceff148 P 6d231fa7f1684535af46a9b196111880 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:45:41.899304 2025 raft_consensus.cc:1203] T da05217d82a74bd6aeb54330c3d6d370 P f7fd7bf659f84ae184696b891db5221a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:45:42.002022 2713 raft_consensus.cc:1050] T afc220634a164930bdaa8ece818fb2e7 P ed5783865eb74d39bee2a38e9139dea4: attempting to promote NON_VOTER 6d231fa7f1684535af46a9b196111880 to VOTER
I20240502 20:45:42.002513 2713 consensus_queue.cc:237] T afc220634a164930bdaa8ece818fb2e7 P ed5783865eb74d39bee2a38e9139dea4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } } peers { permanent_uuid: "6d231fa7f1684535af46a9b196111880" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 43291 } attrs { promote: false } }
I20240502 20:45:42.003672 2428 raft_consensus.cc:1261] T afc220634a164930bdaa8ece818fb2e7 P 6d231fa7f1684535af46a9b196111880 [term 1 LEARNER]: Refusing update from remote peer ed5783865eb74d39bee2a38e9139dea4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:45:42.004109 2953 consensus_queue.cc:1035] T afc220634a164930bdaa8ece818fb2e7 P ed5783865eb74d39bee2a38e9139dea4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d231fa7f1684535af46a9b196111880" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 43291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:45:42.006486 2713 raft_consensus.cc:2932] T afc220634a164930bdaa8ece818fb2e7 P ed5783865eb74d39bee2a38e9139dea4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6d231fa7f1684535af46a9b196111880 (127.30.32.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } } peers { permanent_uuid: "6d231fa7f1684535af46a9b196111880" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 43291 } attrs { promote: false } } }
I20240502 20:45:42.006995 2428 raft_consensus.cc:2932] T afc220634a164930bdaa8ece818fb2e7 P 6d231fa7f1684535af46a9b196111880 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6d231fa7f1684535af46a9b196111880 (127.30.32.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } } peers { permanent_uuid: "6d231fa7f1684535af46a9b196111880" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 43291 } attrs { promote: false } } }
I20240502 20:45:42.008829 744 catalog_manager.cc:5484] T afc220634a164930bdaa8ece818fb2e7 P ed5783865eb74d39bee2a38e9139dea4 reported cstate change: config changed from index 4 to 5, 6d231fa7f1684535af46a9b196111880 (127.30.32.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ed5783865eb74d39bee2a38e9139dea4" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d231fa7f1684535af46a9b196111880" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 43291 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:42.064177 2728 raft_consensus.cc:1050] T 49eb89bf68d942398c68e2bf1d0f6370 P 33be9a5b29214a8d84c2fd7a4936a920: attempting to promote NON_VOTER 0dfc68b63d15462398c76bfbff0e3cf8 to VOTER
I20240502 20:45:42.064646 2728 consensus_queue.cc:237] T 49eb89bf68d942398c68e2bf1d0f6370 P 33be9a5b29214a8d84c2fd7a4936a920 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "0dfc68b63d15462398c76bfbff0e3cf8" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 40439 } attrs { promote: false } }
I20240502 20:45:42.065830 2559 raft_consensus.cc:1261] T 49eb89bf68d942398c68e2bf1d0f6370 P 0dfc68b63d15462398c76bfbff0e3cf8 [term 1 LEARNER]: Refusing update from remote peer 33be9a5b29214a8d84c2fd7a4936a920: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:45:42.066270 2950 consensus_queue.cc:1035] T 49eb89bf68d942398c68e2bf1d0f6370 P 33be9a5b29214a8d84c2fd7a4936a920 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfc68b63d15462398c76bfbff0e3cf8" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:45:42.068681 2728 raft_consensus.cc:2932] T 49eb89bf68d942398c68e2bf1d0f6370 P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0dfc68b63d15462398c76bfbff0e3cf8 (127.30.32.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "0dfc68b63d15462398c76bfbff0e3cf8" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 40439 } attrs { promote: false } } }
I20240502 20:45:42.069190 2559 raft_consensus.cc:2932] T 49eb89bf68d942398c68e2bf1d0f6370 P 0dfc68b63d15462398c76bfbff0e3cf8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0dfc68b63d15462398c76bfbff0e3cf8 (127.30.32.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "0dfc68b63d15462398c76bfbff0e3cf8" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 40439 } attrs { promote: false } } }
I20240502 20:45:42.070991 750 catalog_manager.cc:5484] T 49eb89bf68d942398c68e2bf1d0f6370 P 33be9a5b29214a8d84c2fd7a4936a920 reported cstate change: config changed from index 4 to 5, 0dfc68b63d15462398c76bfbff0e3cf8 (127.30.32.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "33be9a5b29214a8d84c2fd7a4936a920" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dfc68b63d15462398c76bfbff0e3cf8" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 40439 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:42.097731 2728 raft_consensus.cc:1050] T 0734f8b2eeb04ad99ff827c0ecf90663 P 33be9a5b29214a8d84c2fd7a4936a920: attempting to promote NON_VOTER cec988e98db64e2f98b030efc16d4108 to VOTER
I20240502 20:45:42.098049 2728 consensus_queue.cc:237] T 0734f8b2eeb04ad99ff827c0ecf90663 P 33be9a5b29214a8d84c2fd7a4936a920 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "cec988e98db64e2f98b030efc16d4108" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 34789 } attrs { promote: false } }
I20240502 20:45:42.099170 2297 raft_consensus.cc:1261] T 0734f8b2eeb04ad99ff827c0ecf90663 P cec988e98db64e2f98b030efc16d4108 [term 1 LEARNER]: Refusing update from remote peer 33be9a5b29214a8d84c2fd7a4936a920: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:45:42.099611 2728 consensus_queue.cc:1035] T 0734f8b2eeb04ad99ff827c0ecf90663 P 33be9a5b29214a8d84c2fd7a4936a920 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec988e98db64e2f98b030efc16d4108" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 34789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:45:42.102162 2951 raft_consensus.cc:2932] T 0734f8b2eeb04ad99ff827c0ecf90663 P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cec988e98db64e2f98b030efc16d4108 (127.30.32.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "cec988e98db64e2f98b030efc16d4108" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 34789 } attrs { promote: false } } }
I20240502 20:45:42.102703 2297 raft_consensus.cc:2932] T 0734f8b2eeb04ad99ff827c0ecf90663 P cec988e98db64e2f98b030efc16d4108 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cec988e98db64e2f98b030efc16d4108 (127.30.32.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "cec988e98db64e2f98b030efc16d4108" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 34789 } attrs { promote: false } } }
I20240502 20:45:42.105052 744 catalog_manager.cc:5484] T 0734f8b2eeb04ad99ff827c0ecf90663 P 33be9a5b29214a8d84c2fd7a4936a920 reported cstate change: config changed from index 4 to 5, cec988e98db64e2f98b030efc16d4108 (127.30.32.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "33be9a5b29214a8d84c2fd7a4936a920" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec988e98db64e2f98b030efc16d4108" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 34789 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:42.150005 893 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0734f8b2eeb04ad99ff827c0ecf90663"
dest_uuid: "33be9a5b29214a8d84c2fd7a4936a920"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37812
I20240502 20:45:42.150223 893 raft_consensus.cc:595] T 0734f8b2eeb04ad99ff827c0ecf90663 P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 LEADER]: Received request to transfer leadership
I20240502 20:45:42.154036 2953 raft_consensus.cc:1050] T 7a888b6096c2427f8a7f67b26c3c00b7 P ed5783865eb74d39bee2a38e9139dea4: attempting to promote NON_VOTER 0dfc68b63d15462398c76bfbff0e3cf8 to VOTER
I20240502 20:45:42.154413 2953 consensus_queue.cc:237] T 7a888b6096c2427f8a7f67b26c3c00b7 P ed5783865eb74d39bee2a38e9139dea4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } } peers { permanent_uuid: "0dfc68b63d15462398c76bfbff0e3cf8" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 40439 } attrs { promote: false } }
I20240502 20:45:42.155608 2559 raft_consensus.cc:1261] T 7a888b6096c2427f8a7f67b26c3c00b7 P 0dfc68b63d15462398c76bfbff0e3cf8 [term 1 LEARNER]: Refusing update from remote peer ed5783865eb74d39bee2a38e9139dea4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:45:42.156057 2953 consensus_queue.cc:1035] T 7a888b6096c2427f8a7f67b26c3c00b7 P ed5783865eb74d39bee2a38e9139dea4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfc68b63d15462398c76bfbff0e3cf8" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:45:42.157987 2713 raft_consensus.cc:2932] T 7a888b6096c2427f8a7f67b26c3c00b7 P ed5783865eb74d39bee2a38e9139dea4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0dfc68b63d15462398c76bfbff0e3cf8 (127.30.32.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } } peers { permanent_uuid: "0dfc68b63d15462398c76bfbff0e3cf8" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 40439 } attrs { promote: false } } }
I20240502 20:45:42.158480 2559 raft_consensus.cc:2932] T 7a888b6096c2427f8a7f67b26c3c00b7 P 0dfc68b63d15462398c76bfbff0e3cf8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0dfc68b63d15462398c76bfbff0e3cf8 (127.30.32.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } } peers { permanent_uuid: "0dfc68b63d15462398c76bfbff0e3cf8" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 40439 } attrs { promote: false } } }
I20240502 20:45:42.160697 750 catalog_manager.cc:5484] T 7a888b6096c2427f8a7f67b26c3c00b7 P ed5783865eb74d39bee2a38e9139dea4 reported cstate change: config changed from index 4 to 5, 0dfc68b63d15462398c76bfbff0e3cf8 (127.30.32.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ed5783865eb74d39bee2a38e9139dea4" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dfc68b63d15462398c76bfbff0e3cf8" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 40439 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:42.164310 893 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "49eb89bf68d942398c68e2bf1d0f6370"
dest_uuid: "33be9a5b29214a8d84c2fd7a4936a920"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37828
I20240502 20:45:42.164430 893 raft_consensus.cc:595] T 49eb89bf68d942398c68e2bf1d0f6370 P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 LEADER]: Received request to transfer leadership
I20240502 20:45:42.174911 1024 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "afc220634a164930bdaa8ece818fb2e7"
dest_uuid: "ed5783865eb74d39bee2a38e9139dea4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58882
I20240502 20:45:42.175099 1024 raft_consensus.cc:595] T afc220634a164930bdaa8ece818fb2e7 P ed5783865eb74d39bee2a38e9139dea4 [term 1 LEADER]: Received request to transfer leadership
I20240502 20:45:42.194245 1024 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7a888b6096c2427f8a7f67b26c3c00b7"
dest_uuid: "ed5783865eb74d39bee2a38e9139dea4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58892
I20240502 20:45:42.194375 1024 raft_consensus.cc:595] T 7a888b6096c2427f8a7f67b26c3c00b7 P ed5783865eb74d39bee2a38e9139dea4 [term 1 LEADER]: Received request to transfer leadership
I20240502 20:45:42.233237 2950 raft_consensus.cc:1050] T cf7de28bcb20418f99846c121cd88531 P 33be9a5b29214a8d84c2fd7a4936a920: attempting to promote NON_VOTER cec988e98db64e2f98b030efc16d4108 to VOTER
I20240502 20:45:42.233613 2950 consensus_queue.cc:237] T cf7de28bcb20418f99846c121cd88531 P 33be9a5b29214a8d84c2fd7a4936a920 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "cec988e98db64e2f98b030efc16d4108" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 34789 } attrs { promote: false } }
I20240502 20:45:42.234762 2297 raft_consensus.cc:1261] T cf7de28bcb20418f99846c121cd88531 P cec988e98db64e2f98b030efc16d4108 [term 1 LEARNER]: Refusing update from remote peer 33be9a5b29214a8d84c2fd7a4936a920: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:45:42.235222 2950 consensus_queue.cc:1035] T cf7de28bcb20418f99846c121cd88531 P 33be9a5b29214a8d84c2fd7a4936a920 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec988e98db64e2f98b030efc16d4108" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 34789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:45:42.236114 2950 raft_consensus.cc:1050] T 21a27f13d4104a8a87aa8b935c6c8dfa P 33be9a5b29214a8d84c2fd7a4936a920: attempting to promote NON_VOTER 2260afde4ccb498189d7b25c00a2c727 to VOTER
I20240502 20:45:42.236395 2950 consensus_queue.cc:237] T 21a27f13d4104a8a87aa8b935c6c8dfa P 33be9a5b29214a8d84c2fd7a4936a920 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "2260afde4ccb498189d7b25c00a2c727" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 40723 } attrs { promote: false } }
I20240502 20:45:42.237268 2728 raft_consensus.cc:2932] T cf7de28bcb20418f99846c121cd88531 P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cec988e98db64e2f98b030efc16d4108 (127.30.32.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "cec988e98db64e2f98b030efc16d4108" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 34789 } attrs { promote: false } } }
I20240502 20:45:42.237512 2157 raft_consensus.cc:1261] T 21a27f13d4104a8a87aa8b935c6c8dfa P 2260afde4ccb498189d7b25c00a2c727 [term 1 LEARNER]: Refusing update from remote peer 33be9a5b29214a8d84c2fd7a4936a920: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:45:42.237890 2951 consensus_queue.cc:1035] T 21a27f13d4104a8a87aa8b935c6c8dfa P 33be9a5b29214a8d84c2fd7a4936a920 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2260afde4ccb498189d7b25c00a2c727" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 40723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:45:42.239873 2951 raft_consensus.cc:2932] T 21a27f13d4104a8a87aa8b935c6c8dfa P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2260afde4ccb498189d7b25c00a2c727 (127.30.32.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "2260afde4ccb498189d7b25c00a2c727" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 40723 } attrs { promote: false } } }
I20240502 20:45:42.240141 2297 raft_consensus.cc:2932] T cf7de28bcb20418f99846c121cd88531 P cec988e98db64e2f98b030efc16d4108 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cec988e98db64e2f98b030efc16d4108 (127.30.32.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "cec988e98db64e2f98b030efc16d4108" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 34789 } attrs { promote: false } } }
I20240502 20:45:42.240298 744 catalog_manager.cc:5484] T cf7de28bcb20418f99846c121cd88531 P 33be9a5b29214a8d84c2fd7a4936a920 reported cstate change: config changed from index 4 to 5, cec988e98db64e2f98b030efc16d4108 (127.30.32.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "33be9a5b29214a8d84c2fd7a4936a920" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec988e98db64e2f98b030efc16d4108" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 34789 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:42.241709 2157 raft_consensus.cc:2932] T 21a27f13d4104a8a87aa8b935c6c8dfa P 2260afde4ccb498189d7b25c00a2c727 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2260afde4ccb498189d7b25c00a2c727 (127.30.32.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "2260afde4ccb498189d7b25c00a2c727" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 40723 } attrs { promote: false } } }
I20240502 20:45:42.243853 750 catalog_manager.cc:5484] T 21a27f13d4104a8a87aa8b935c6c8dfa P 33be9a5b29214a8d84c2fd7a4936a920 reported cstate change: config changed from index 4 to 5, 2260afde4ccb498189d7b25c00a2c727 (127.30.32.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "33be9a5b29214a8d84c2fd7a4936a920" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2260afde4ccb498189d7b25c00a2c727" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 40723 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:42.329329 2952 raft_consensus.cc:1050] T a6c9b839b6a346aca42a280aed623913 P ed5783865eb74d39bee2a38e9139dea4: attempting to promote NON_VOTER f7fd7bf659f84ae184696b891db5221a to VOTER
I20240502 20:45:42.329702 2952 consensus_queue.cc:237] T a6c9b839b6a346aca42a280aed623913 P ed5783865eb74d39bee2a38e9139dea4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } } peers { permanent_uuid: "f7fd7bf659f84ae184696b891db5221a" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42407 } attrs { promote: false } }
I20240502 20:45:42.331050 2025 raft_consensus.cc:1261] T a6c9b839b6a346aca42a280aed623913 P f7fd7bf659f84ae184696b891db5221a [term 1 LEARNER]: Refusing update from remote peer ed5783865eb74d39bee2a38e9139dea4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:45:42.331478 2713 consensus_queue.cc:1035] T a6c9b839b6a346aca42a280aed623913 P ed5783865eb74d39bee2a38e9139dea4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7fd7bf659f84ae184696b891db5221a" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:45:42.334062 2952 raft_consensus.cc:2932] T a6c9b839b6a346aca42a280aed623913 P ed5783865eb74d39bee2a38e9139dea4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f7fd7bf659f84ae184696b891db5221a (127.30.32.67) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } } peers { permanent_uuid: "f7fd7bf659f84ae184696b891db5221a" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42407 } attrs { promote: false } } }
I20240502 20:45:42.334582 2025 raft_consensus.cc:2932] T a6c9b839b6a346aca42a280aed623913 P f7fd7bf659f84ae184696b891db5221a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f7fd7bf659f84ae184696b891db5221a (127.30.32.67) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } } peers { permanent_uuid: "f7fd7bf659f84ae184696b891db5221a" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42407 } attrs { promote: false } } }
I20240502 20:45:42.336843 744 catalog_manager.cc:5484] T a6c9b839b6a346aca42a280aed623913 P ed5783865eb74d39bee2a38e9139dea4 reported cstate change: config changed from index 4 to 5, f7fd7bf659f84ae184696b891db5221a (127.30.32.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ed5783865eb74d39bee2a38e9139dea4" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7fd7bf659f84ae184696b891db5221a" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42407 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:42.363169 2728 raft_consensus.cc:1050] T da05217d82a74bd6aeb54330c3d6d370 P 33be9a5b29214a8d84c2fd7a4936a920: attempting to promote NON_VOTER f7fd7bf659f84ae184696b891db5221a to VOTER
I20240502 20:45:42.363525 2728 consensus_queue.cc:237] T da05217d82a74bd6aeb54330c3d6d370 P 33be9a5b29214a8d84c2fd7a4936a920 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "f7fd7bf659f84ae184696b891db5221a" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42407 } attrs { promote: false } }
I20240502 20:45:42.364660 2025 raft_consensus.cc:1261] T da05217d82a74bd6aeb54330c3d6d370 P f7fd7bf659f84ae184696b891db5221a [term 1 LEARNER]: Refusing update from remote peer 33be9a5b29214a8d84c2fd7a4936a920: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:45:42.365123 2728 consensus_queue.cc:1035] T da05217d82a74bd6aeb54330c3d6d370 P 33be9a5b29214a8d84c2fd7a4936a920 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7fd7bf659f84ae184696b891db5221a" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:45:42.366788 2950 raft_consensus.cc:2932] T da05217d82a74bd6aeb54330c3d6d370 P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f7fd7bf659f84ae184696b891db5221a (127.30.32.67) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "f7fd7bf659f84ae184696b891db5221a" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42407 } attrs { promote: false } } }
I20240502 20:45:42.367250 2025 raft_consensus.cc:2932] T da05217d82a74bd6aeb54330c3d6d370 P f7fd7bf659f84ae184696b891db5221a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f7fd7bf659f84ae184696b891db5221a (127.30.32.67) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "f7fd7bf659f84ae184696b891db5221a" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42407 } attrs { promote: false } } }
I20240502 20:45:42.369535 750 catalog_manager.cc:5484] T da05217d82a74bd6aeb54330c3d6d370 P 33be9a5b29214a8d84c2fd7a4936a920 reported cstate change: config changed from index 4 to 5, f7fd7bf659f84ae184696b891db5221a (127.30.32.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "33be9a5b29214a8d84c2fd7a4936a920" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7fd7bf659f84ae184696b891db5221a" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42407 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:42.377727 2952 raft_consensus.cc:1050] T 74b7ff29e276451985e7cbe72ceff148 P ed5783865eb74d39bee2a38e9139dea4: attempting to promote NON_VOTER 6d231fa7f1684535af46a9b196111880 to VOTER
I20240502 20:45:42.378088 2952 consensus_queue.cc:237] T 74b7ff29e276451985e7cbe72ceff148 P ed5783865eb74d39bee2a38e9139dea4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } } peers { permanent_uuid: "6d231fa7f1684535af46a9b196111880" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 43291 } attrs { promote: false } }
I20240502 20:45:42.379144 2428 raft_consensus.cc:1261] T 74b7ff29e276451985e7cbe72ceff148 P 6d231fa7f1684535af46a9b196111880 [term 1 LEARNER]: Refusing update from remote peer ed5783865eb74d39bee2a38e9139dea4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:45:42.379534 2952 consensus_queue.cc:1035] T 74b7ff29e276451985e7cbe72ceff148 P ed5783865eb74d39bee2a38e9139dea4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d231fa7f1684535af46a9b196111880" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 43291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:45:42.381484 2713 raft_consensus.cc:2932] T 74b7ff29e276451985e7cbe72ceff148 P ed5783865eb74d39bee2a38e9139dea4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6d231fa7f1684535af46a9b196111880 (127.30.32.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } } peers { permanent_uuid: "6d231fa7f1684535af46a9b196111880" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 43291 } attrs { promote: false } } }
I20240502 20:45:42.382004 2428 raft_consensus.cc:2932] T 74b7ff29e276451985e7cbe72ceff148 P 6d231fa7f1684535af46a9b196111880 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6d231fa7f1684535af46a9b196111880 (127.30.32.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } } peers { permanent_uuid: "6d231fa7f1684535af46a9b196111880" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 43291 } attrs { promote: false } } }
I20240502 20:45:42.384163 744 catalog_manager.cc:5484] T 74b7ff29e276451985e7cbe72ceff148 P ed5783865eb74d39bee2a38e9139dea4 reported cstate change: config changed from index 4 to 5, 6d231fa7f1684535af46a9b196111880 (127.30.32.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ed5783865eb74d39bee2a38e9139dea4" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d231fa7f1684535af46a9b196111880" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 43291 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:42.405371 2953 raft_consensus.cc:979] T afc220634a164930bdaa8ece818fb2e7 P ed5783865eb74d39bee2a38e9139dea4: : Instructing follower 6d231fa7f1684535af46a9b196111880 to start an election
I20240502 20:45:42.405520 2953 raft_consensus.cc:1067] T afc220634a164930bdaa8ece818fb2e7 P ed5783865eb74d39bee2a38e9139dea4 [term 1 LEADER]: Signalling peer 6d231fa7f1684535af46a9b196111880 to start an election
I20240502 20:45:42.405675 893 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "da05217d82a74bd6aeb54330c3d6d370"
dest_uuid: "33be9a5b29214a8d84c2fd7a4936a920"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37848
I20240502 20:45:42.405818 893 raft_consensus.cc:595] T da05217d82a74bd6aeb54330c3d6d370 P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 LEADER]: Received request to transfer leadership
I20240502 20:45:42.406174 2428 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "afc220634a164930bdaa8ece818fb2e7"
dest_uuid: "6d231fa7f1684535af46a9b196111880"
from {username='slave'} at 127.30.32.66:51213
I20240502 20:45:42.406352 2428 raft_consensus.cc:482] T afc220634a164930bdaa8ece818fb2e7 P 6d231fa7f1684535af46a9b196111880 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:45:42.406466 2428 raft_consensus.cc:3037] T afc220634a164930bdaa8ece818fb2e7 P 6d231fa7f1684535af46a9b196111880 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:42.407326 2428 raft_consensus.cc:504] T afc220634a164930bdaa8ece818fb2e7 P 6d231fa7f1684535af46a9b196111880 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } } peers { permanent_uuid: "6d231fa7f1684535af46a9b196111880" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 43291 } attrs { promote: false } }
I20240502 20:45:42.408226 2428 leader_election.cc:288] T afc220634a164930bdaa8ece818fb2e7 P 6d231fa7f1684535af46a9b196111880 [CANDIDATE]: Term 2 election: Requested vote from peers ed5783865eb74d39bee2a38e9139dea4 (127.30.32.66:45113)
I20240502 20:45:42.408689 1024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afc220634a164930bdaa8ece818fb2e7" candidate_uuid: "6d231fa7f1684535af46a9b196111880" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "ed5783865eb74d39bee2a38e9139dea4"
I20240502 20:45:42.408906 1024 raft_consensus.cc:3032] T afc220634a164930bdaa8ece818fb2e7 P ed5783865eb74d39bee2a38e9139dea4 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:45:42.408973 1024 raft_consensus.cc:729] T afc220634a164930bdaa8ece818fb2e7 P ed5783865eb74d39bee2a38e9139dea4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ed5783865eb74d39bee2a38e9139dea4, State: Running, Role: LEADER
I20240502 20:45:42.409368 1024 consensus_queue.cc:260] T afc220634a164930bdaa8ece818fb2e7 P ed5783865eb74d39bee2a38e9139dea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } } peers { permanent_uuid: "6d231fa7f1684535af46a9b196111880" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 43291 } attrs { promote: false } }
I20240502 20:45:42.409526 1024 raft_consensus.cc:3037] T afc220634a164930bdaa8ece818fb2e7 P ed5783865eb74d39bee2a38e9139dea4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:42.410430 1024 raft_consensus.cc:2454] T afc220634a164930bdaa8ece818fb2e7 P ed5783865eb74d39bee2a38e9139dea4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d231fa7f1684535af46a9b196111880 in term 2.
I20240502 20:45:42.410910 2364 leader_election.cc:302] T afc220634a164930bdaa8ece818fb2e7 P 6d231fa7f1684535af46a9b196111880 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6d231fa7f1684535af46a9b196111880, ed5783865eb74d39bee2a38e9139dea4; no voters:
I20240502 20:45:42.411162 2871 raft_consensus.cc:2781] T afc220634a164930bdaa8ece818fb2e7 P 6d231fa7f1684535af46a9b196111880 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:45:42.411360 2871 raft_consensus.cc:686] T afc220634a164930bdaa8ece818fb2e7 P 6d231fa7f1684535af46a9b196111880 [term 2 LEADER]: Becoming Leader. State: Replica: 6d231fa7f1684535af46a9b196111880, State: Running, Role: LEADER
I20240502 20:45:42.411667 2871 consensus_queue.cc:237] T afc220634a164930bdaa8ece818fb2e7 P 6d231fa7f1684535af46a9b196111880 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } } peers { permanent_uuid: "6d231fa7f1684535af46a9b196111880" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 43291 } attrs { promote: false } }
I20240502 20:45:42.413601 750 catalog_manager.cc:5484] T afc220634a164930bdaa8ece818fb2e7 P 6d231fa7f1684535af46a9b196111880 reported cstate change: term changed from 1 to 2, leader changed from ed5783865eb74d39bee2a38e9139dea4 (127.30.32.66) to 6d231fa7f1684535af46a9b196111880 (127.30.32.70). New cstate: current_term: 2 leader_uuid: "6d231fa7f1684535af46a9b196111880" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d231fa7f1684535af46a9b196111880" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 43291 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 20:45:42.420634 736 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet afc220634a164930bdaa8ece818fb2e7 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 6d231fa7f1684535af46a9b196111880 (127.30.32.70:43291) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20240502 20:45:42.423085 1024 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "74b7ff29e276451985e7cbe72ceff148"
dest_uuid: "ed5783865eb74d39bee2a38e9139dea4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58912
I20240502 20:45:42.423240 1024 raft_consensus.cc:595] T 74b7ff29e276451985e7cbe72ceff148 P ed5783865eb74d39bee2a38e9139dea4 [term 1 LEADER]: Received request to transfer leadership
I20240502 20:45:42.433828 893 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cf7de28bcb20418f99846c121cd88531"
dest_uuid: "33be9a5b29214a8d84c2fd7a4936a920"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37864
I20240502 20:45:42.433980 893 raft_consensus.cc:595] T cf7de28bcb20418f99846c121cd88531 P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 LEADER]: Received request to transfer leadership
I20240502 20:45:42.443550 1024 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a6c9b839b6a346aca42a280aed623913"
dest_uuid: "ed5783865eb74d39bee2a38e9139dea4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58930
I20240502 20:45:42.443686 1024 raft_consensus.cc:595] T a6c9b839b6a346aca42a280aed623913 P ed5783865eb74d39bee2a38e9139dea4 [term 1 LEADER]: Received request to transfer leadership
I20240502 20:45:42.453434 893 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0734f8b2eeb04ad99ff827c0ecf90663"
dest_uuid: "33be9a5b29214a8d84c2fd7a4936a920"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37880
I20240502 20:45:42.453576 893 raft_consensus.cc:595] T 0734f8b2eeb04ad99ff827c0ecf90663 P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 LEADER]: Received request to transfer leadership
I20240502 20:45:42.463933 893 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "49eb89bf68d942398c68e2bf1d0f6370"
dest_uuid: "33be9a5b29214a8d84c2fd7a4936a920"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37894
I20240502 20:45:42.464062 893 raft_consensus.cc:595] T 49eb89bf68d942398c68e2bf1d0f6370 P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 LEADER]: Received request to transfer leadership
I20240502 20:45:42.479427 893 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "21a27f13d4104a8a87aa8b935c6c8dfa"
dest_uuid: "33be9a5b29214a8d84c2fd7a4936a920"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37914
I20240502 20:45:42.479568 893 raft_consensus.cc:595] T 21a27f13d4104a8a87aa8b935c6c8dfa P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 LEADER]: Received request to transfer leadership
I20240502 20:45:42.489286 1024 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7a888b6096c2427f8a7f67b26c3c00b7"
dest_uuid: "ed5783865eb74d39bee2a38e9139dea4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58950
I20240502 20:45:42.489421 1024 raft_consensus.cc:595] T 7a888b6096c2427f8a7f67b26c3c00b7 P ed5783865eb74d39bee2a38e9139dea4 [term 1 LEADER]: Received request to transfer leadership
I20240502 20:45:42.542255 2951 raft_consensus.cc:979] T 0734f8b2eeb04ad99ff827c0ecf90663 P 33be9a5b29214a8d84c2fd7a4936a920: : Instructing follower cec988e98db64e2f98b030efc16d4108 to start an election
I20240502 20:45:42.542379 2951 raft_consensus.cc:1067] T 0734f8b2eeb04ad99ff827c0ecf90663 P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 LEADER]: Signalling peer cec988e98db64e2f98b030efc16d4108 to start an election
I20240502 20:45:42.542970 2297 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0734f8b2eeb04ad99ff827c0ecf90663"
dest_uuid: "cec988e98db64e2f98b030efc16d4108"
from {username='slave'} at 127.30.32.65:36405
I20240502 20:45:42.543155 2297 raft_consensus.cc:482] T 0734f8b2eeb04ad99ff827c0ecf90663 P cec988e98db64e2f98b030efc16d4108 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:45:42.543236 2297 raft_consensus.cc:3037] T 0734f8b2eeb04ad99ff827c0ecf90663 P cec988e98db64e2f98b030efc16d4108 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:42.544045 2297 raft_consensus.cc:504] T 0734f8b2eeb04ad99ff827c0ecf90663 P cec988e98db64e2f98b030efc16d4108 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "cec988e98db64e2f98b030efc16d4108" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 34789 } attrs { promote: false } }
I20240502 20:45:42.544863 2297 leader_election.cc:288] T 0734f8b2eeb04ad99ff827c0ecf90663 P cec988e98db64e2f98b030efc16d4108 [CANDIDATE]: Term 2 election: Requested vote from peers 33be9a5b29214a8d84c2fd7a4936a920 (127.30.32.65:45537)
I20240502 20:45:42.545337 893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0734f8b2eeb04ad99ff827c0ecf90663" candidate_uuid: "cec988e98db64e2f98b030efc16d4108" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "33be9a5b29214a8d84c2fd7a4936a920"
I20240502 20:45:42.545482 893 raft_consensus.cc:3032] T 0734f8b2eeb04ad99ff827c0ecf90663 P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:45:42.545519 893 raft_consensus.cc:729] T 0734f8b2eeb04ad99ff827c0ecf90663 P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 33be9a5b29214a8d84c2fd7a4936a920, State: Running, Role: LEADER
I20240502 20:45:42.545644 893 consensus_queue.cc:260] T 0734f8b2eeb04ad99ff827c0ecf90663 P 33be9a5b29214a8d84c2fd7a4936a920 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "cec988e98db64e2f98b030efc16d4108" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 34789 } attrs { promote: false } }
I20240502 20:45:42.545768 893 raft_consensus.cc:3037] T 0734f8b2eeb04ad99ff827c0ecf90663 P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:42.546538 893 raft_consensus.cc:2454] T 0734f8b2eeb04ad99ff827c0ecf90663 P 33be9a5b29214a8d84c2fd7a4936a920 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec988e98db64e2f98b030efc16d4108 in term 2.
I20240502 20:45:42.546883 2230 leader_election.cc:302] T 0734f8b2eeb04ad99ff827c0ecf90663 P cec988e98db64e2f98b030efc16d4108 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 33be9a5b29214a8d84c2fd7a4936a920, cec988e98db64e2f98b030efc16d4108; no voters:
I20240502 20:45:42.547106 2852 raft_consensus.cc:2781] T 0734f8b2eeb04ad99ff827c0ecf90663 P cec988e98db64e2f98b030efc16d4108 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:45:42.547283 2852 raft_consensus.cc:686] T 0734f8b2eeb04ad99ff827c0ecf90663 P cec988e98db64e2f98b030efc16d4108 [term 2 LEADER]: Becoming Leader. State: Replica: cec988e98db64e2f98b030efc16d4108, State: Running, Role: LEADER
I20240502 20:45:42.547535 2852 consensus_queue.cc:237] T 0734f8b2eeb04ad99ff827c0ecf90663 P cec988e98db64e2f98b030efc16d4108 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "cec988e98db64e2f98b030efc16d4108" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 34789 } attrs { promote: false } }
I20240502 20:45:42.549546 750 catalog_manager.cc:5484] T 0734f8b2eeb04ad99ff827c0ecf90663 P cec988e98db64e2f98b030efc16d4108 reported cstate change: term changed from 1 to 2, leader changed from 33be9a5b29214a8d84c2fd7a4936a920 (127.30.32.65) to cec988e98db64e2f98b030efc16d4108 (127.30.32.69). New cstate: current_term: 2 leader_uuid: "cec988e98db64e2f98b030efc16d4108" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec988e98db64e2f98b030efc16d4108" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 34789 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:42.588611 2950 raft_consensus.cc:979] T 49eb89bf68d942398c68e2bf1d0f6370 P 33be9a5b29214a8d84c2fd7a4936a920: : Instructing follower 0dfc68b63d15462398c76bfbff0e3cf8 to start an election
I20240502 20:45:42.588745 2950 raft_consensus.cc:1067] T 49eb89bf68d942398c68e2bf1d0f6370 P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 LEADER]: Signalling peer 0dfc68b63d15462398c76bfbff0e3cf8 to start an election
I20240502 20:45:42.589231 2559 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "49eb89bf68d942398c68e2bf1d0f6370"
dest_uuid: "0dfc68b63d15462398c76bfbff0e3cf8"
from {username='slave'} at 127.30.32.65:43153
I20240502 20:45:42.589440 2559 raft_consensus.cc:482] T 49eb89bf68d942398c68e2bf1d0f6370 P 0dfc68b63d15462398c76bfbff0e3cf8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:45:42.589548 2559 raft_consensus.cc:3037] T 49eb89bf68d942398c68e2bf1d0f6370 P 0dfc68b63d15462398c76bfbff0e3cf8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:42.590404 2559 raft_consensus.cc:504] T 49eb89bf68d942398c68e2bf1d0f6370 P 0dfc68b63d15462398c76bfbff0e3cf8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "0dfc68b63d15462398c76bfbff0e3cf8" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 40439 } attrs { promote: false } }
I20240502 20:45:42.591289 2559 leader_election.cc:288] T 49eb89bf68d942398c68e2bf1d0f6370 P 0dfc68b63d15462398c76bfbff0e3cf8 [CANDIDATE]: Term 2 election: Requested vote from peers 33be9a5b29214a8d84c2fd7a4936a920 (127.30.32.65:45537)
I20240502 20:45:42.591715 893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49eb89bf68d942398c68e2bf1d0f6370" candidate_uuid: "0dfc68b63d15462398c76bfbff0e3cf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "33be9a5b29214a8d84c2fd7a4936a920"
I20240502 20:45:42.591833 893 raft_consensus.cc:3032] T 49eb89bf68d942398c68e2bf1d0f6370 P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:45:42.591876 893 raft_consensus.cc:729] T 49eb89bf68d942398c68e2bf1d0f6370 P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 33be9a5b29214a8d84c2fd7a4936a920, State: Running, Role: LEADER
I20240502 20:45:42.592041 893 consensus_queue.cc:260] T 49eb89bf68d942398c68e2bf1d0f6370 P 33be9a5b29214a8d84c2fd7a4936a920 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "0dfc68b63d15462398c76bfbff0e3cf8" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 40439 } attrs { promote: false } }
I20240502 20:45:42.592175 893 raft_consensus.cc:3037] T 49eb89bf68d942398c68e2bf1d0f6370 P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:42.593000 893 raft_consensus.cc:2454] T 49eb89bf68d942398c68e2bf1d0f6370 P 33be9a5b29214a8d84c2fd7a4936a920 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dfc68b63d15462398c76bfbff0e3cf8 in term 2.
I20240502 20:45:42.593370 2492 leader_election.cc:302] T 49eb89bf68d942398c68e2bf1d0f6370 P 0dfc68b63d15462398c76bfbff0e3cf8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0dfc68b63d15462398c76bfbff0e3cf8, 33be9a5b29214a8d84c2fd7a4936a920; no voters:
I20240502 20:45:42.593606 2872 raft_consensus.cc:2781] T 49eb89bf68d942398c68e2bf1d0f6370 P 0dfc68b63d15462398c76bfbff0e3cf8 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:45:42.593786 2872 raft_consensus.cc:686] T 49eb89bf68d942398c68e2bf1d0f6370 P 0dfc68b63d15462398c76bfbff0e3cf8 [term 2 LEADER]: Becoming Leader. State: Replica: 0dfc68b63d15462398c76bfbff0e3cf8, State: Running, Role: LEADER
I20240502 20:45:42.594038 2872 consensus_queue.cc:237] T 49eb89bf68d942398c68e2bf1d0f6370 P 0dfc68b63d15462398c76bfbff0e3cf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "0dfc68b63d15462398c76bfbff0e3cf8" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 40439 } attrs { promote: false } }
I20240502 20:45:42.595877 750 catalog_manager.cc:5484] T 49eb89bf68d942398c68e2bf1d0f6370 P 0dfc68b63d15462398c76bfbff0e3cf8 reported cstate change: term changed from 1 to 2, leader changed from 33be9a5b29214a8d84c2fd7a4936a920 (127.30.32.65) to 0dfc68b63d15462398c76bfbff0e3cf8 (127.30.32.71). New cstate: current_term: 2 leader_uuid: "0dfc68b63d15462398c76bfbff0e3cf8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dfc68b63d15462398c76bfbff0e3cf8" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 40439 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:42.655907 2953 raft_consensus.cc:979] T 7a888b6096c2427f8a7f67b26c3c00b7 P ed5783865eb74d39bee2a38e9139dea4: : Instructing follower 0dfc68b63d15462398c76bfbff0e3cf8 to start an election
I20240502 20:45:42.656020 2953 raft_consensus.cc:1067] T 7a888b6096c2427f8a7f67b26c3c00b7 P ed5783865eb74d39bee2a38e9139dea4 [term 1 LEADER]: Signalling peer 0dfc68b63d15462398c76bfbff0e3cf8 to start an election
I20240502 20:45:42.656518 2559 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7a888b6096c2427f8a7f67b26c3c00b7"
dest_uuid: "0dfc68b63d15462398c76bfbff0e3cf8"
from {username='slave'} at 127.30.32.66:47783
I20240502 20:45:42.656651 2559 raft_consensus.cc:482] T 7a888b6096c2427f8a7f67b26c3c00b7 P 0dfc68b63d15462398c76bfbff0e3cf8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:45:42.656699 2559 raft_consensus.cc:3037] T 7a888b6096c2427f8a7f67b26c3c00b7 P 0dfc68b63d15462398c76bfbff0e3cf8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:42.657527 2559 raft_consensus.cc:504] T 7a888b6096c2427f8a7f67b26c3c00b7 P 0dfc68b63d15462398c76bfbff0e3cf8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } } peers { permanent_uuid: "0dfc68b63d15462398c76bfbff0e3cf8" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 40439 } attrs { promote: false } }
I20240502 20:45:42.657852 2559 leader_election.cc:288] T 7a888b6096c2427f8a7f67b26c3c00b7 P 0dfc68b63d15462398c76bfbff0e3cf8 [CANDIDATE]: Term 2 election: Requested vote from peers ed5783865eb74d39bee2a38e9139dea4 (127.30.32.66:45113)
I20240502 20:45:42.658252 1024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a888b6096c2427f8a7f67b26c3c00b7" candidate_uuid: "0dfc68b63d15462398c76bfbff0e3cf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "ed5783865eb74d39bee2a38e9139dea4"
I20240502 20:45:42.658377 1024 raft_consensus.cc:3032] T 7a888b6096c2427f8a7f67b26c3c00b7 P ed5783865eb74d39bee2a38e9139dea4 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:45:42.658414 1024 raft_consensus.cc:729] T 7a888b6096c2427f8a7f67b26c3c00b7 P ed5783865eb74d39bee2a38e9139dea4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ed5783865eb74d39bee2a38e9139dea4, State: Running, Role: LEADER
I20240502 20:45:42.658529 1024 consensus_queue.cc:260] T 7a888b6096c2427f8a7f67b26c3c00b7 P ed5783865eb74d39bee2a38e9139dea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } } peers { permanent_uuid: "0dfc68b63d15462398c76bfbff0e3cf8" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 40439 } attrs { promote: false } }
I20240502 20:45:42.658643 1024 raft_consensus.cc:3037] T 7a888b6096c2427f8a7f67b26c3c00b7 P ed5783865eb74d39bee2a38e9139dea4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:42.659355 1024 raft_consensus.cc:2454] T 7a888b6096c2427f8a7f67b26c3c00b7 P ed5783865eb74d39bee2a38e9139dea4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dfc68b63d15462398c76bfbff0e3cf8 in term 2.
I20240502 20:45:42.659669 2495 leader_election.cc:302] T 7a888b6096c2427f8a7f67b26c3c00b7 P 0dfc68b63d15462398c76bfbff0e3cf8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0dfc68b63d15462398c76bfbff0e3cf8, ed5783865eb74d39bee2a38e9139dea4; no voters:
I20240502 20:45:42.659868 2872 raft_consensus.cc:2781] T 7a888b6096c2427f8a7f67b26c3c00b7 P 0dfc68b63d15462398c76bfbff0e3cf8 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:45:42.659967 2872 raft_consensus.cc:686] T 7a888b6096c2427f8a7f67b26c3c00b7 P 0dfc68b63d15462398c76bfbff0e3cf8 [term 2 LEADER]: Becoming Leader. State: Replica: 0dfc68b63d15462398c76bfbff0e3cf8, State: Running, Role: LEADER
I20240502 20:45:42.660104 2872 consensus_queue.cc:237] T 7a888b6096c2427f8a7f67b26c3c00b7 P 0dfc68b63d15462398c76bfbff0e3cf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } } peers { permanent_uuid: "0dfc68b63d15462398c76bfbff0e3cf8" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 40439 } attrs { promote: false } }
I20240502 20:45:42.661599 750 catalog_manager.cc:5484] T 7a888b6096c2427f8a7f67b26c3c00b7 P 0dfc68b63d15462398c76bfbff0e3cf8 reported cstate change: term changed from 1 to 2, leader changed from ed5783865eb74d39bee2a38e9139dea4 (127.30.32.66) to 0dfc68b63d15462398c76bfbff0e3cf8 (127.30.32.71). New cstate: current_term: 2 leader_uuid: "0dfc68b63d15462398c76bfbff0e3cf8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dfc68b63d15462398c76bfbff0e3cf8" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 40439 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:42.700213 893 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "da05217d82a74bd6aeb54330c3d6d370"
dest_uuid: "33be9a5b29214a8d84c2fd7a4936a920"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37922
I20240502 20:45:42.700367 893 raft_consensus.cc:595] T da05217d82a74bd6aeb54330c3d6d370 P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 LEADER]: Received request to transfer leadership
I20240502 20:45:42.710269 1024 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "74b7ff29e276451985e7cbe72ceff148"
dest_uuid: "ed5783865eb74d39bee2a38e9139dea4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58964
I20240502 20:45:42.710402 1024 raft_consensus.cc:595] T 74b7ff29e276451985e7cbe72ceff148 P ed5783865eb74d39bee2a38e9139dea4 [term 1 LEADER]: Received request to transfer leadership
I20240502 20:45:42.719717 893 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cf7de28bcb20418f99846c121cd88531"
dest_uuid: "33be9a5b29214a8d84c2fd7a4936a920"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37938
I20240502 20:45:42.719863 893 raft_consensus.cc:595] T cf7de28bcb20418f99846c121cd88531 P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 LEADER]: Received request to transfer leadership
I20240502 20:45:42.731943 1024 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a6c9b839b6a346aca42a280aed623913"
dest_uuid: "ed5783865eb74d39bee2a38e9139dea4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58982
I20240502 20:45:42.732066 1024 raft_consensus.cc:595] T a6c9b839b6a346aca42a280aed623913 P ed5783865eb74d39bee2a38e9139dea4 [term 1 LEADER]: Received request to transfer leadership
I20240502 20:45:42.741525 2951 raft_consensus.cc:979] T cf7de28bcb20418f99846c121cd88531 P 33be9a5b29214a8d84c2fd7a4936a920: : Instructing follower cec988e98db64e2f98b030efc16d4108 to start an election
I20240502 20:45:42.741648 2951 raft_consensus.cc:1067] T cf7de28bcb20418f99846c121cd88531 P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 LEADER]: Signalling peer cec988e98db64e2f98b030efc16d4108 to start an election
I20240502 20:45:42.742077 2297 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cf7de28bcb20418f99846c121cd88531"
dest_uuid: "cec988e98db64e2f98b030efc16d4108"
from {username='slave'} at 127.30.32.65:36405
I20240502 20:45:42.742210 2297 raft_consensus.cc:482] T cf7de28bcb20418f99846c121cd88531 P cec988e98db64e2f98b030efc16d4108 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:45:42.742280 2297 raft_consensus.cc:3037] T cf7de28bcb20418f99846c121cd88531 P cec988e98db64e2f98b030efc16d4108 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:42.743047 2297 raft_consensus.cc:504] T cf7de28bcb20418f99846c121cd88531 P cec988e98db64e2f98b030efc16d4108 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "cec988e98db64e2f98b030efc16d4108" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 34789 } attrs { promote: false } }
I20240502 20:45:42.743372 2297 leader_election.cc:288] T cf7de28bcb20418f99846c121cd88531 P cec988e98db64e2f98b030efc16d4108 [CANDIDATE]: Term 2 election: Requested vote from peers 33be9a5b29214a8d84c2fd7a4936a920 (127.30.32.65:45537)
I20240502 20:45:42.743724 893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf7de28bcb20418f99846c121cd88531" candidate_uuid: "cec988e98db64e2f98b030efc16d4108" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "33be9a5b29214a8d84c2fd7a4936a920"
I20240502 20:45:42.743836 893 raft_consensus.cc:3032] T cf7de28bcb20418f99846c121cd88531 P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:45:42.743871 893 raft_consensus.cc:729] T cf7de28bcb20418f99846c121cd88531 P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 33be9a5b29214a8d84c2fd7a4936a920, State: Running, Role: LEADER
I20240502 20:45:42.743988 893 consensus_queue.cc:260] T cf7de28bcb20418f99846c121cd88531 P 33be9a5b29214a8d84c2fd7a4936a920 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "cec988e98db64e2f98b030efc16d4108" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 34789 } attrs { promote: false } }
I20240502 20:45:42.744094 893 raft_consensus.cc:3037] T cf7de28bcb20418f99846c121cd88531 P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:42.744791 893 raft_consensus.cc:2454] T cf7de28bcb20418f99846c121cd88531 P 33be9a5b29214a8d84c2fd7a4936a920 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec988e98db64e2f98b030efc16d4108 in term 2.
I20240502 20:45:42.745105 2230 leader_election.cc:302] T cf7de28bcb20418f99846c121cd88531 P cec988e98db64e2f98b030efc16d4108 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 33be9a5b29214a8d84c2fd7a4936a920, cec988e98db64e2f98b030efc16d4108; no voters:
I20240502 20:45:42.745352 2852 raft_consensus.cc:2781] T cf7de28bcb20418f99846c121cd88531 P cec988e98db64e2f98b030efc16d4108 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:45:42.745457 2852 raft_consensus.cc:686] T cf7de28bcb20418f99846c121cd88531 P cec988e98db64e2f98b030efc16d4108 [term 2 LEADER]: Becoming Leader. State: Replica: cec988e98db64e2f98b030efc16d4108, State: Running, Role: LEADER
I20240502 20:45:42.745586 2852 consensus_queue.cc:237] T cf7de28bcb20418f99846c121cd88531 P cec988e98db64e2f98b030efc16d4108 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "cec988e98db64e2f98b030efc16d4108" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 34789 } attrs { promote: false } }
I20240502 20:45:42.747135 750 catalog_manager.cc:5484] T cf7de28bcb20418f99846c121cd88531 P cec988e98db64e2f98b030efc16d4108 reported cstate change: term changed from 1 to 2, leader changed from 33be9a5b29214a8d84c2fd7a4936a920 (127.30.32.65) to cec988e98db64e2f98b030efc16d4108 (127.30.32.69). New cstate: current_term: 2 leader_uuid: "cec988e98db64e2f98b030efc16d4108" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec988e98db64e2f98b030efc16d4108" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 34789 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:42.760756 893 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "21a27f13d4104a8a87aa8b935c6c8dfa"
dest_uuid: "33be9a5b29214a8d84c2fd7a4936a920"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37962
I20240502 20:45:42.760916 893 raft_consensus.cc:595] T 21a27f13d4104a8a87aa8b935c6c8dfa P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 LEADER]: Received request to transfer leadership
I20240502 20:45:42.782510 2950 raft_consensus.cc:979] T 21a27f13d4104a8a87aa8b935c6c8dfa P 33be9a5b29214a8d84c2fd7a4936a920: : Instructing follower 2260afde4ccb498189d7b25c00a2c727 to start an election
I20240502 20:45:42.782624 2950 raft_consensus.cc:1067] T 21a27f13d4104a8a87aa8b935c6c8dfa P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 LEADER]: Signalling peer 2260afde4ccb498189d7b25c00a2c727 to start an election
I20240502 20:45:42.783103 2157 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "21a27f13d4104a8a87aa8b935c6c8dfa"
dest_uuid: "2260afde4ccb498189d7b25c00a2c727"
from {username='slave'} at 127.30.32.65:43031
I20240502 20:45:42.783298 2157 raft_consensus.cc:482] T 21a27f13d4104a8a87aa8b935c6c8dfa P 2260afde4ccb498189d7b25c00a2c727 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:45:42.783365 2157 raft_consensus.cc:3037] T 21a27f13d4104a8a87aa8b935c6c8dfa P 2260afde4ccb498189d7b25c00a2c727 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:42.784164 2157 raft_consensus.cc:504] T 21a27f13d4104a8a87aa8b935c6c8dfa P 2260afde4ccb498189d7b25c00a2c727 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "2260afde4ccb498189d7b25c00a2c727" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 40723 } attrs { promote: false } }
I20240502 20:45:42.784893 2157 leader_election.cc:288] T 21a27f13d4104a8a87aa8b935c6c8dfa P 2260afde4ccb498189d7b25c00a2c727 [CANDIDATE]: Term 2 election: Requested vote from peers 33be9a5b29214a8d84c2fd7a4936a920 (127.30.32.65:45537)
I20240502 20:45:42.785427 893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21a27f13d4104a8a87aa8b935c6c8dfa" candidate_uuid: "2260afde4ccb498189d7b25c00a2c727" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "33be9a5b29214a8d84c2fd7a4936a920"
I20240502 20:45:42.785538 893 raft_consensus.cc:3032] T 21a27f13d4104a8a87aa8b935c6c8dfa P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:45:42.785588 893 raft_consensus.cc:729] T 21a27f13d4104a8a87aa8b935c6c8dfa P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 33be9a5b29214a8d84c2fd7a4936a920, State: Running, Role: LEADER
I20240502 20:45:42.785768 893 consensus_queue.cc:260] T 21a27f13d4104a8a87aa8b935c6c8dfa P 33be9a5b29214a8d84c2fd7a4936a920 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "2260afde4ccb498189d7b25c00a2c727" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 40723 } attrs { promote: false } }
I20240502 20:45:42.785907 893 raft_consensus.cc:3037] T 21a27f13d4104a8a87aa8b935c6c8dfa P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:42.786746 893 raft_consensus.cc:2454] T 21a27f13d4104a8a87aa8b935c6c8dfa P 33be9a5b29214a8d84c2fd7a4936a920 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2260afde4ccb498189d7b25c00a2c727 in term 2.
I20240502 20:45:42.787073 2089 leader_election.cc:302] T 21a27f13d4104a8a87aa8b935c6c8dfa P 2260afde4ccb498189d7b25c00a2c727 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2260afde4ccb498189d7b25c00a2c727, 33be9a5b29214a8d84c2fd7a4936a920; no voters:
I20240502 20:45:42.787372 3294 raft_consensus.cc:2781] T 21a27f13d4104a8a87aa8b935c6c8dfa P 2260afde4ccb498189d7b25c00a2c727 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:45:42.787528 3294 raft_consensus.cc:686] T 21a27f13d4104a8a87aa8b935c6c8dfa P 2260afde4ccb498189d7b25c00a2c727 [term 2 LEADER]: Becoming Leader. State: Replica: 2260afde4ccb498189d7b25c00a2c727, State: Running, Role: LEADER
I20240502 20:45:42.787669 3294 consensus_queue.cc:237] T 21a27f13d4104a8a87aa8b935c6c8dfa P 2260afde4ccb498189d7b25c00a2c727 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "2260afde4ccb498189d7b25c00a2c727" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 40723 } attrs { promote: false } }
I20240502 20:45:42.789523 750 catalog_manager.cc:5484] T 21a27f13d4104a8a87aa8b935c6c8dfa P 2260afde4ccb498189d7b25c00a2c727 reported cstate change: term changed from 1 to 2, leader changed from 33be9a5b29214a8d84c2fd7a4936a920 (127.30.32.65) to 2260afde4ccb498189d7b25c00a2c727 (127.30.32.68). New cstate: current_term: 2 leader_uuid: "2260afde4ccb498189d7b25c00a2c727" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2260afde4ccb498189d7b25c00a2c727" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 40723 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:42.856676 2951 raft_consensus.cc:979] T da05217d82a74bd6aeb54330c3d6d370 P 33be9a5b29214a8d84c2fd7a4936a920: : Instructing follower f7fd7bf659f84ae184696b891db5221a to start an election
I20240502 20:45:42.856799 2951 raft_consensus.cc:1067] T da05217d82a74bd6aeb54330c3d6d370 P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 LEADER]: Signalling peer f7fd7bf659f84ae184696b891db5221a to start an election
I20240502 20:45:42.857293 2025 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "da05217d82a74bd6aeb54330c3d6d370"
dest_uuid: "f7fd7bf659f84ae184696b891db5221a"
from {username='slave'} at 127.30.32.65:52051
I20240502 20:45:42.857483 2025 raft_consensus.cc:482] T da05217d82a74bd6aeb54330c3d6d370 P f7fd7bf659f84ae184696b891db5221a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:45:42.857599 2025 raft_consensus.cc:3037] T da05217d82a74bd6aeb54330c3d6d370 P f7fd7bf659f84ae184696b891db5221a [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:42.858484 2025 raft_consensus.cc:504] T da05217d82a74bd6aeb54330c3d6d370 P f7fd7bf659f84ae184696b891db5221a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "f7fd7bf659f84ae184696b891db5221a" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42407 } attrs { promote: false } }
I20240502 20:45:42.859357 2025 leader_election.cc:288] T da05217d82a74bd6aeb54330c3d6d370 P f7fd7bf659f84ae184696b891db5221a [CANDIDATE]: Term 2 election: Requested vote from peers 33be9a5b29214a8d84c2fd7a4936a920 (127.30.32.65:45537)
I20240502 20:45:42.859783 893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05217d82a74bd6aeb54330c3d6d370" candidate_uuid: "f7fd7bf659f84ae184696b891db5221a" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "33be9a5b29214a8d84c2fd7a4936a920"
I20240502 20:45:42.859898 893 raft_consensus.cc:3032] T da05217d82a74bd6aeb54330c3d6d370 P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:45:42.859951 893 raft_consensus.cc:729] T da05217d82a74bd6aeb54330c3d6d370 P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 33be9a5b29214a8d84c2fd7a4936a920, State: Running, Role: LEADER
I20240502 20:45:42.860111 893 consensus_queue.cc:260] T da05217d82a74bd6aeb54330c3d6d370 P 33be9a5b29214a8d84c2fd7a4936a920 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "f7fd7bf659f84ae184696b891db5221a" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42407 } attrs { promote: false } }
I20240502 20:45:42.860262 893 raft_consensus.cc:3037] T da05217d82a74bd6aeb54330c3d6d370 P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:42.861116 893 raft_consensus.cc:2454] T da05217d82a74bd6aeb54330c3d6d370 P 33be9a5b29214a8d84c2fd7a4936a920 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7fd7bf659f84ae184696b891db5221a in term 2.
I20240502 20:45:42.861483 1958 leader_election.cc:302] T da05217d82a74bd6aeb54330c3d6d370 P f7fd7bf659f84ae184696b891db5221a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 33be9a5b29214a8d84c2fd7a4936a920, f7fd7bf659f84ae184696b891db5221a; no voters:
I20240502 20:45:42.861707 2891 raft_consensus.cc:2781] T da05217d82a74bd6aeb54330c3d6d370 P f7fd7bf659f84ae184696b891db5221a [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:45:42.861925 2891 raft_consensus.cc:686] T da05217d82a74bd6aeb54330c3d6d370 P f7fd7bf659f84ae184696b891db5221a [term 2 LEADER]: Becoming Leader. State: Replica: f7fd7bf659f84ae184696b891db5221a, State: Running, Role: LEADER
I20240502 20:45:42.862203 2891 consensus_queue.cc:237] T da05217d82a74bd6aeb54330c3d6d370 P f7fd7bf659f84ae184696b891db5221a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "f7fd7bf659f84ae184696b891db5221a" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42407 } attrs { promote: false } }
I20240502 20:45:42.864137 750 catalog_manager.cc:5484] T da05217d82a74bd6aeb54330c3d6d370 P f7fd7bf659f84ae184696b891db5221a reported cstate change: term changed from 1 to 2, leader changed from 33be9a5b29214a8d84c2fd7a4936a920 (127.30.32.65) to f7fd7bf659f84ae184696b891db5221a (127.30.32.67). New cstate: current_term: 2 leader_uuid: "f7fd7bf659f84ae184696b891db5221a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7fd7bf659f84ae184696b891db5221a" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42407 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:42.893256 2713 raft_consensus.cc:979] T a6c9b839b6a346aca42a280aed623913 P ed5783865eb74d39bee2a38e9139dea4: : Instructing follower f7fd7bf659f84ae184696b891db5221a to start an election
I20240502 20:45:42.893388 2713 raft_consensus.cc:1067] T a6c9b839b6a346aca42a280aed623913 P ed5783865eb74d39bee2a38e9139dea4 [term 1 LEADER]: Signalling peer f7fd7bf659f84ae184696b891db5221a to start an election
I20240502 20:45:42.893848 2025 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a6c9b839b6a346aca42a280aed623913"
dest_uuid: "f7fd7bf659f84ae184696b891db5221a"
from {username='slave'} at 127.30.32.66:48027
I20240502 20:45:42.893990 2025 raft_consensus.cc:482] T a6c9b839b6a346aca42a280aed623913 P f7fd7bf659f84ae184696b891db5221a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:45:42.894055 2025 raft_consensus.cc:3037] T a6c9b839b6a346aca42a280aed623913 P f7fd7bf659f84ae184696b891db5221a [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:42.894860 2025 raft_consensus.cc:504] T a6c9b839b6a346aca42a280aed623913 P f7fd7bf659f84ae184696b891db5221a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } } peers { permanent_uuid: "f7fd7bf659f84ae184696b891db5221a" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42407 } attrs { promote: false } }
I20240502 20:45:42.895220 2025 leader_election.cc:288] T a6c9b839b6a346aca42a280aed623913 P f7fd7bf659f84ae184696b891db5221a [CANDIDATE]: Term 2 election: Requested vote from peers ed5783865eb74d39bee2a38e9139dea4 (127.30.32.66:45113)
I20240502 20:45:42.895589 1024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6c9b839b6a346aca42a280aed623913" candidate_uuid: "f7fd7bf659f84ae184696b891db5221a" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "ed5783865eb74d39bee2a38e9139dea4"
I20240502 20:45:42.895695 1024 raft_consensus.cc:3032] T a6c9b839b6a346aca42a280aed623913 P ed5783865eb74d39bee2a38e9139dea4 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:45:42.895758 1024 raft_consensus.cc:729] T a6c9b839b6a346aca42a280aed623913 P ed5783865eb74d39bee2a38e9139dea4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ed5783865eb74d39bee2a38e9139dea4, State: Running, Role: LEADER
I20240502 20:45:42.895910 1024 consensus_queue.cc:260] T a6c9b839b6a346aca42a280aed623913 P ed5783865eb74d39bee2a38e9139dea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } } peers { permanent_uuid: "f7fd7bf659f84ae184696b891db5221a" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42407 } attrs { promote: false } }
I20240502 20:45:42.896032 1024 raft_consensus.cc:3037] T a6c9b839b6a346aca42a280aed623913 P ed5783865eb74d39bee2a38e9139dea4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:42.896735 1024 raft_consensus.cc:2454] T a6c9b839b6a346aca42a280aed623913 P ed5783865eb74d39bee2a38e9139dea4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7fd7bf659f84ae184696b891db5221a in term 2.
I20240502 20:45:42.897038 1961 leader_election.cc:302] T a6c9b839b6a346aca42a280aed623913 P f7fd7bf659f84ae184696b891db5221a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: ed5783865eb74d39bee2a38e9139dea4, f7fd7bf659f84ae184696b891db5221a; no voters:
I20240502 20:45:42.897214 2891 raft_consensus.cc:2781] T a6c9b839b6a346aca42a280aed623913 P f7fd7bf659f84ae184696b891db5221a [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:45:42.897310 2891 raft_consensus.cc:686] T a6c9b839b6a346aca42a280aed623913 P f7fd7bf659f84ae184696b891db5221a [term 2 LEADER]: Becoming Leader. State: Replica: f7fd7bf659f84ae184696b891db5221a, State: Running, Role: LEADER
I20240502 20:45:42.897449 2891 consensus_queue.cc:237] T a6c9b839b6a346aca42a280aed623913 P f7fd7bf659f84ae184696b891db5221a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } } peers { permanent_uuid: "f7fd7bf659f84ae184696b891db5221a" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42407 } attrs { promote: false } }
I20240502 20:45:42.898877 750 catalog_manager.cc:5484] T a6c9b839b6a346aca42a280aed623913 P f7fd7bf659f84ae184696b891db5221a reported cstate change: term changed from 1 to 2, leader changed from ed5783865eb74d39bee2a38e9139dea4 (127.30.32.66) to f7fd7bf659f84ae184696b891db5221a (127.30.32.67). New cstate: current_term: 2 leader_uuid: "f7fd7bf659f84ae184696b891db5221a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7fd7bf659f84ae184696b891db5221a" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42407 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:42.909310 2953 raft_consensus.cc:979] T 74b7ff29e276451985e7cbe72ceff148 P ed5783865eb74d39bee2a38e9139dea4: : Instructing follower 6d231fa7f1684535af46a9b196111880 to start an election
I20240502 20:45:42.909397 2953 raft_consensus.cc:1067] T 74b7ff29e276451985e7cbe72ceff148 P ed5783865eb74d39bee2a38e9139dea4 [term 1 LEADER]: Signalling peer 6d231fa7f1684535af46a9b196111880 to start an election
I20240502 20:45:42.909824 2428 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "74b7ff29e276451985e7cbe72ceff148"
dest_uuid: "6d231fa7f1684535af46a9b196111880"
from {username='slave'} at 127.30.32.66:51213
I20240502 20:45:42.909950 2428 raft_consensus.cc:482] T 74b7ff29e276451985e7cbe72ceff148 P 6d231fa7f1684535af46a9b196111880 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:45:42.910017 2428 raft_consensus.cc:3037] T 74b7ff29e276451985e7cbe72ceff148 P 6d231fa7f1684535af46a9b196111880 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:42.910838 2428 raft_consensus.cc:504] T 74b7ff29e276451985e7cbe72ceff148 P 6d231fa7f1684535af46a9b196111880 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } } peers { permanent_uuid: "6d231fa7f1684535af46a9b196111880" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 43291 } attrs { promote: false } }
I20240502 20:45:42.911115 2428 leader_election.cc:288] T 74b7ff29e276451985e7cbe72ceff148 P 6d231fa7f1684535af46a9b196111880 [CANDIDATE]: Term 2 election: Requested vote from peers ed5783865eb74d39bee2a38e9139dea4 (127.30.32.66:45113)
I20240502 20:45:42.911604 1024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74b7ff29e276451985e7cbe72ceff148" candidate_uuid: "6d231fa7f1684535af46a9b196111880" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "ed5783865eb74d39bee2a38e9139dea4"
I20240502 20:45:42.911716 1024 raft_consensus.cc:3032] T 74b7ff29e276451985e7cbe72ceff148 P ed5783865eb74d39bee2a38e9139dea4 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:45:42.911760 1024 raft_consensus.cc:729] T 74b7ff29e276451985e7cbe72ceff148 P ed5783865eb74d39bee2a38e9139dea4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ed5783865eb74d39bee2a38e9139dea4, State: Running, Role: LEADER
I20240502 20:45:42.911900 1024 consensus_queue.cc:260] T 74b7ff29e276451985e7cbe72ceff148 P ed5783865eb74d39bee2a38e9139dea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } } peers { permanent_uuid: "6d231fa7f1684535af46a9b196111880" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 43291 } attrs { promote: false } }
I20240502 20:45:42.912019 1024 raft_consensus.cc:3037] T 74b7ff29e276451985e7cbe72ceff148 P ed5783865eb74d39bee2a38e9139dea4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:42.912770 1024 raft_consensus.cc:2454] T 74b7ff29e276451985e7cbe72ceff148 P ed5783865eb74d39bee2a38e9139dea4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d231fa7f1684535af46a9b196111880 in term 2.
I20240502 20:45:42.913056 2364 leader_election.cc:302] T 74b7ff29e276451985e7cbe72ceff148 P 6d231fa7f1684535af46a9b196111880 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6d231fa7f1684535af46a9b196111880, ed5783865eb74d39bee2a38e9139dea4; no voters:
I20240502 20:45:42.913331 3299 raft_consensus.cc:2781] T 74b7ff29e276451985e7cbe72ceff148 P 6d231fa7f1684535af46a9b196111880 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:45:42.913465 3299 raft_consensus.cc:686] T 74b7ff29e276451985e7cbe72ceff148 P 6d231fa7f1684535af46a9b196111880 [term 2 LEADER]: Becoming Leader. State: Replica: 6d231fa7f1684535af46a9b196111880, State: Running, Role: LEADER
I20240502 20:45:42.913591 3299 consensus_queue.cc:237] T 74b7ff29e276451985e7cbe72ceff148 P 6d231fa7f1684535af46a9b196111880 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } } peers { permanent_uuid: "6d231fa7f1684535af46a9b196111880" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 43291 } attrs { promote: false } }
I20240502 20:45:42.914991 750 catalog_manager.cc:5484] T 74b7ff29e276451985e7cbe72ceff148 P 6d231fa7f1684535af46a9b196111880 reported cstate change: term changed from 1 to 2, leader changed from ed5783865eb74d39bee2a38e9139dea4 (127.30.32.66) to 6d231fa7f1684535af46a9b196111880 (127.30.32.70). New cstate: current_term: 2 leader_uuid: "6d231fa7f1684535af46a9b196111880" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d231fa7f1684535af46a9b196111880" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 43291 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:43.009292 893 raft_consensus.cc:1261] T 49eb89bf68d942398c68e2bf1d0f6370 P 33be9a5b29214a8d84c2fd7a4936a920 [term 2 FOLLOWER]: Refusing update from remote peer 0dfc68b63d15462398c76bfbff0e3cf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 20:45:43.009904 2872 consensus_queue.cc:1035] T 49eb89bf68d942398c68e2bf1d0f6370 P 0dfc68b63d15462398c76bfbff0e3cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:45:43.016633 1024 raft_consensus.cc:1261] T afc220634a164930bdaa8ece818fb2e7 P ed5783865eb74d39bee2a38e9139dea4 [term 2 FOLLOWER]: Refusing update from remote peer 6d231fa7f1684535af46a9b196111880: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 20:45:43.017271 3299 consensus_queue.cc:1035] T afc220634a164930bdaa8ece818fb2e7 P 6d231fa7f1684535af46a9b196111880 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:45:43.028541 893 raft_consensus.cc:1261] T 0734f8b2eeb04ad99ff827c0ecf90663 P 33be9a5b29214a8d84c2fd7a4936a920 [term 2 FOLLOWER]: Refusing update from remote peer cec988e98db64e2f98b030efc16d4108: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 20:45:43.029085 2852 consensus_queue.cc:1035] T 0734f8b2eeb04ad99ff827c0ecf90663 P cec988e98db64e2f98b030efc16d4108 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:45:43.064368 2428 consensus_queue.cc:237] T afc220634a164930bdaa8ece818fb2e7 P 6d231fa7f1684535af46a9b196111880 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d231fa7f1684535af46a9b196111880" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 43291 } attrs { promote: false } }
I20240502 20:45:43.065025 3299 raft_consensus.cc:2932] T afc220634a164930bdaa8ece818fb2e7 P 6d231fa7f1684535af46a9b196111880 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER ed5783865eb74d39bee2a38e9139dea4 (127.30.32.66) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d231fa7f1684535af46a9b196111880" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 43291 } attrs { promote: false } } }
I20240502 20:45:43.066278 736 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet afc220634a164930bdaa8ece818fb2e7 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 20:45:43.067179 744 catalog_manager.cc:5484] T afc220634a164930bdaa8ece818fb2e7 P 6d231fa7f1684535af46a9b196111880 reported cstate change: config changed from index 5 to 7, VOTER ed5783865eb74d39bee2a38e9139dea4 (127.30.32.66) evicted. New cstate: current_term: 2 leader_uuid: "6d231fa7f1684535af46a9b196111880" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d231fa7f1684535af46a9b196111880" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 43291 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:43.069536 1003 tablet_service.cc:1514] Processing DeleteTablet for tablet afc220634a164930bdaa8ece818fb2e7 with delete_type TABLET_DATA_TOMBSTONED (TS ed5783865eb74d39bee2a38e9139dea4 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57638
I20240502 20:45:43.070051 3352 tablet_replica.cc:330] T afc220634a164930bdaa8ece818fb2e7 P ed5783865eb74d39bee2a38e9139dea4: stopping tablet replica
I20240502 20:45:43.070379 3352 raft_consensus.cc:2229] T afc220634a164930bdaa8ece818fb2e7 P ed5783865eb74d39bee2a38e9139dea4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:43.070550 3352 raft_consensus.cc:2258] T afc220634a164930bdaa8ece818fb2e7 P ed5783865eb74d39bee2a38e9139dea4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:43.071571 3352 ts_tablet_manager.cc:1875] T afc220634a164930bdaa8ece818fb2e7 P ed5783865eb74d39bee2a38e9139dea4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:45:43.073454 3352 ts_tablet_manager.cc:1888] T afc220634a164930bdaa8ece818fb2e7 P ed5783865eb74d39bee2a38e9139dea4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 20:45:43.073531 3352 log.cc:1196] T afc220634a164930bdaa8ece818fb2e7 P ed5783865eb74d39bee2a38e9139dea4: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal/wals/afc220634a164930bdaa8ece818fb2e7
I20240502 20:45:43.074074 738 catalog_manager.cc:4830] TS ed5783865eb74d39bee2a38e9139dea4 (127.30.32.66:45113): tablet afc220634a164930bdaa8ece818fb2e7 (table rebalance_test_table_3 [id=2a2a74b349514a8aa4c6c02c2533a51f]) successfully deleted
I20240502 20:45:43.148532 893 raft_consensus.cc:1261] T cf7de28bcb20418f99846c121cd88531 P 33be9a5b29214a8d84c2fd7a4936a920 [term 2 FOLLOWER]: Refusing update from remote peer cec988e98db64e2f98b030efc16d4108: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 20:45:43.149118 3220 consensus_queue.cc:1035] T cf7de28bcb20418f99846c121cd88531 P cec988e98db64e2f98b030efc16d4108 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:45:43.199092 2297 consensus_queue.cc:237] T 0734f8b2eeb04ad99ff827c0ecf90663 P cec988e98db64e2f98b030efc16d4108 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cec988e98db64e2f98b030efc16d4108" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 34789 } attrs { promote: false } }
I20240502 20:45:43.199802 3220 raft_consensus.cc:2932] T 0734f8b2eeb04ad99ff827c0ecf90663 P cec988e98db64e2f98b030efc16d4108 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 33be9a5b29214a8d84c2fd7a4936a920 (127.30.32.65) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cec988e98db64e2f98b030efc16d4108" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 34789 } attrs { promote: false } } }
I20240502 20:45:43.201234 737 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0734f8b2eeb04ad99ff827c0ecf90663 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 20:45:43.202162 744 catalog_manager.cc:5484] T 0734f8b2eeb04ad99ff827c0ecf90663 P cec988e98db64e2f98b030efc16d4108 reported cstate change: config changed from index 5 to 7, VOTER 33be9a5b29214a8d84c2fd7a4936a920 (127.30.32.65) evicted. New cstate: current_term: 2 leader_uuid: "cec988e98db64e2f98b030efc16d4108" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cec988e98db64e2f98b030efc16d4108" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 34789 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:43.204608 873 tablet_service.cc:1514] Processing DeleteTablet for tablet 0734f8b2eeb04ad99ff827c0ecf90663 with delete_type TABLET_DATA_TOMBSTONED (TS 33be9a5b29214a8d84c2fd7a4936a920 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:54242
I20240502 20:45:43.205142 3353 tablet_replica.cc:330] T 0734f8b2eeb04ad99ff827c0ecf90663 P 33be9a5b29214a8d84c2fd7a4936a920: stopping tablet replica
I20240502 20:45:43.205425 3353 raft_consensus.cc:2229] T 0734f8b2eeb04ad99ff827c0ecf90663 P 33be9a5b29214a8d84c2fd7a4936a920 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:43.205590 3353 raft_consensus.cc:2258] T 0734f8b2eeb04ad99ff827c0ecf90663 P 33be9a5b29214a8d84c2fd7a4936a920 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:43.206547 3353 ts_tablet_manager.cc:1875] T 0734f8b2eeb04ad99ff827c0ecf90663 P 33be9a5b29214a8d84c2fd7a4936a920: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:45:43.208401 3353 ts_tablet_manager.cc:1888] T 0734f8b2eeb04ad99ff827c0ecf90663 P 33be9a5b29214a8d84c2fd7a4936a920: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 20:45:43.208478 3353 log.cc:1196] T 0734f8b2eeb04ad99ff827c0ecf90663 P 33be9a5b29214a8d84c2fd7a4936a920: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal/wals/0734f8b2eeb04ad99ff827c0ecf90663
I20240502 20:45:43.209103 735 catalog_manager.cc:4830] TS 33be9a5b29214a8d84c2fd7a4936a920 (127.30.32.65:45537): tablet 0734f8b2eeb04ad99ff827c0ecf90663 (table rebalance_test_table_4 [id=1c8687d7588245508f9c47ea918ebc6a]) successfully deleted
I20240502 20:45:43.229861 1024 raft_consensus.cc:1261] T 7a888b6096c2427f8a7f67b26c3c00b7 P ed5783865eb74d39bee2a38e9139dea4 [term 2 FOLLOWER]: Refusing update from remote peer 0dfc68b63d15462398c76bfbff0e3cf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 20:45:43.230396 2872 consensus_queue.cc:1035] T 7a888b6096c2427f8a7f67b26c3c00b7 P 0dfc68b63d15462398c76bfbff0e3cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:45:43.247387 2559 consensus_queue.cc:237] T 7a888b6096c2427f8a7f67b26c3c00b7 P 0dfc68b63d15462398c76bfbff0e3cf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0dfc68b63d15462398c76bfbff0e3cf8" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 40439 } attrs { promote: false } }
I20240502 20:45:43.248037 3222 raft_consensus.cc:2932] T 7a888b6096c2427f8a7f67b26c3c00b7 P 0dfc68b63d15462398c76bfbff0e3cf8 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER ed5783865eb74d39bee2a38e9139dea4 (127.30.32.66) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0dfc68b63d15462398c76bfbff0e3cf8" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 40439 } attrs { promote: false } } }
I20240502 20:45:43.250018 738 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7a888b6096c2427f8a7f67b26c3c00b7 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 20:45:43.250866 744 catalog_manager.cc:5484] T 7a888b6096c2427f8a7f67b26c3c00b7 P 0dfc68b63d15462398c76bfbff0e3cf8 reported cstate change: config changed from index 5 to 7, VOTER ed5783865eb74d39bee2a38e9139dea4 (127.30.32.66) evicted. New cstate: current_term: 2 leader_uuid: "0dfc68b63d15462398c76bfbff0e3cf8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0dfc68b63d15462398c76bfbff0e3cf8" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 40439 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:43.253285 1003 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a888b6096c2427f8a7f67b26c3c00b7 with delete_type TABLET_DATA_TOMBSTONED (TS ed5783865eb74d39bee2a38e9139dea4 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57638
I20240502 20:45:43.253511 3352 tablet_replica.cc:330] T 7a888b6096c2427f8a7f67b26c3c00b7 P ed5783865eb74d39bee2a38e9139dea4: stopping tablet replica
I20240502 20:45:43.253641 3352 raft_consensus.cc:2229] T 7a888b6096c2427f8a7f67b26c3c00b7 P ed5783865eb74d39bee2a38e9139dea4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:43.253747 3352 raft_consensus.cc:2258] T 7a888b6096c2427f8a7f67b26c3c00b7 P ed5783865eb74d39bee2a38e9139dea4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:43.254397 3352 ts_tablet_manager.cc:1875] T 7a888b6096c2427f8a7f67b26c3c00b7 P ed5783865eb74d39bee2a38e9139dea4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:45:43.256198 3352 ts_tablet_manager.cc:1888] T 7a888b6096c2427f8a7f67b26c3c00b7 P ed5783865eb74d39bee2a38e9139dea4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 20:45:43.256285 3352 log.cc:1196] T 7a888b6096c2427f8a7f67b26c3c00b7 P ed5783865eb74d39bee2a38e9139dea4: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal/wals/7a888b6096c2427f8a7f67b26c3c00b7
I20240502 20:45:43.256753 738 catalog_manager.cc:4830] TS ed5783865eb74d39bee2a38e9139dea4 (127.30.32.66:45113): tablet 7a888b6096c2427f8a7f67b26c3c00b7 (table rebalance_test_table_1 [id=da321cf3a7944f5ba716ef84b816582e]) successfully deleted
I20240502 20:45:43.258553 893 raft_consensus.cc:1261] T 21a27f13d4104a8a87aa8b935c6c8dfa P 33be9a5b29214a8d84c2fd7a4936a920 [term 2 FOLLOWER]: Refusing update from remote peer 2260afde4ccb498189d7b25c00a2c727: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 20:45:43.259022 3294 consensus_queue.cc:1035] T 21a27f13d4104a8a87aa8b935c6c8dfa P 2260afde4ccb498189d7b25c00a2c727 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:45:43.264772 2559 consensus_queue.cc:237] T 49eb89bf68d942398c68e2bf1d0f6370 P 0dfc68b63d15462398c76bfbff0e3cf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0dfc68b63d15462398c76bfbff0e3cf8" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 40439 } attrs { promote: false } }
I20240502 20:45:43.265359 3222 raft_consensus.cc:2932] T 49eb89bf68d942398c68e2bf1d0f6370 P 0dfc68b63d15462398c76bfbff0e3cf8 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 33be9a5b29214a8d84c2fd7a4936a920 (127.30.32.65) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0dfc68b63d15462398c76bfbff0e3cf8" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 40439 } attrs { promote: false } } }
I20240502 20:45:43.266551 738 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 49eb89bf68d942398c68e2bf1d0f6370 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 20:45:43.267388 750 catalog_manager.cc:5484] T 49eb89bf68d942398c68e2bf1d0f6370 P 0dfc68b63d15462398c76bfbff0e3cf8 reported cstate change: config changed from index 5 to 7, VOTER 33be9a5b29214a8d84c2fd7a4936a920 (127.30.32.65) evicted. New cstate: current_term: 2 leader_uuid: "0dfc68b63d15462398c76bfbff0e3cf8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0dfc68b63d15462398c76bfbff0e3cf8" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 40439 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:43.269629 873 tablet_service.cc:1514] Processing DeleteTablet for tablet 49eb89bf68d942398c68e2bf1d0f6370 with delete_type TABLET_DATA_TOMBSTONED (TS 33be9a5b29214a8d84c2fd7a4936a920 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:54242
I20240502 20:45:43.270385 3353 tablet_replica.cc:330] T 49eb89bf68d942398c68e2bf1d0f6370 P 33be9a5b29214a8d84c2fd7a4936a920: stopping tablet replica
I20240502 20:45:43.270534 3353 raft_consensus.cc:2229] T 49eb89bf68d942398c68e2bf1d0f6370 P 33be9a5b29214a8d84c2fd7a4936a920 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:43.270656 3353 raft_consensus.cc:2258] T 49eb89bf68d942398c68e2bf1d0f6370 P 33be9a5b29214a8d84c2fd7a4936a920 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:43.271353 3353 ts_tablet_manager.cc:1875] T 49eb89bf68d942398c68e2bf1d0f6370 P 33be9a5b29214a8d84c2fd7a4936a920: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:45:43.273451 3353 ts_tablet_manager.cc:1888] T 49eb89bf68d942398c68e2bf1d0f6370 P 33be9a5b29214a8d84c2fd7a4936a920: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 20:45:43.273535 3353 log.cc:1196] T 49eb89bf68d942398c68e2bf1d0f6370 P 33be9a5b29214a8d84c2fd7a4936a920: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal/wals/49eb89bf68d942398c68e2bf1d0f6370
I20240502 20:45:43.274256 735 catalog_manager.cc:4830] TS 33be9a5b29214a8d84c2fd7a4936a920 (127.30.32.65:45537): tablet 49eb89bf68d942398c68e2bf1d0f6370 (table rebalance_test_table_2 [id=fddf24b281f54ee394d7a819baadce47]) successfully deleted
I20240502 20:45:43.330202 893 raft_consensus.cc:1261] T da05217d82a74bd6aeb54330c3d6d370 P 33be9a5b29214a8d84c2fd7a4936a920 [term 2 FOLLOWER]: Refusing update from remote peer f7fd7bf659f84ae184696b891db5221a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 20:45:43.331086 2891 consensus_queue.cc:1035] T da05217d82a74bd6aeb54330c3d6d370 P f7fd7bf659f84ae184696b891db5221a [LEADER]: Connected to new peer: Peer: permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:45:43.376307 2297 consensus_queue.cc:237] T cf7de28bcb20418f99846c121cd88531 P cec988e98db64e2f98b030efc16d4108 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cec988e98db64e2f98b030efc16d4108" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 34789 } attrs { promote: false } }
I20240502 20:45:43.376969 3220 raft_consensus.cc:2932] T cf7de28bcb20418f99846c121cd88531 P cec988e98db64e2f98b030efc16d4108 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 33be9a5b29214a8d84c2fd7a4936a920 (127.30.32.65) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cec988e98db64e2f98b030efc16d4108" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 34789 } attrs { promote: false } } }
I20240502 20:45:43.378310 737 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cf7de28bcb20418f99846c121cd88531 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 20:45:43.379268 750 catalog_manager.cc:5484] T cf7de28bcb20418f99846c121cd88531 P cec988e98db64e2f98b030efc16d4108 reported cstate change: config changed from index 5 to 7, VOTER 33be9a5b29214a8d84c2fd7a4936a920 (127.30.32.65) evicted. New cstate: current_term: 2 leader_uuid: "cec988e98db64e2f98b030efc16d4108" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cec988e98db64e2f98b030efc16d4108" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 34789 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:43.381609 873 tablet_service.cc:1514] Processing DeleteTablet for tablet cf7de28bcb20418f99846c121cd88531 with delete_type TABLET_DATA_TOMBSTONED (TS 33be9a5b29214a8d84c2fd7a4936a920 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:54242
I20240502 20:45:43.381835 3353 tablet_replica.cc:330] T cf7de28bcb20418f99846c121cd88531 P 33be9a5b29214a8d84c2fd7a4936a920: stopping tablet replica
I20240502 20:45:43.381963 3353 raft_consensus.cc:2229] T cf7de28bcb20418f99846c121cd88531 P 33be9a5b29214a8d84c2fd7a4936a920 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:43.382054 3353 raft_consensus.cc:2258] T cf7de28bcb20418f99846c121cd88531 P 33be9a5b29214a8d84c2fd7a4936a920 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:43.382601 3353 ts_tablet_manager.cc:1875] T cf7de28bcb20418f99846c121cd88531 P 33be9a5b29214a8d84c2fd7a4936a920: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:45:43.384361 3353 ts_tablet_manager.cc:1888] T cf7de28bcb20418f99846c121cd88531 P 33be9a5b29214a8d84c2fd7a4936a920: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 20:45:43.384423 3353 log.cc:1196] T cf7de28bcb20418f99846c121cd88531 P 33be9a5b29214a8d84c2fd7a4936a920: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal/wals/cf7de28bcb20418f99846c121cd88531
I20240502 20:45:43.384877 735 catalog_manager.cc:4830] TS 33be9a5b29214a8d84c2fd7a4936a920 (127.30.32.65:45537): tablet cf7de28bcb20418f99846c121cd88531 (table rebalance_test_table_0 [id=9738245a8d55447f94af91ee9ca2ce96]) successfully deleted
I20240502 20:45:43.426306 1024 raft_consensus.cc:1261] T 74b7ff29e276451985e7cbe72ceff148 P ed5783865eb74d39bee2a38e9139dea4 [term 2 FOLLOWER]: Refusing update from remote peer 6d231fa7f1684535af46a9b196111880: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 20:45:43.426844 3300 consensus_queue.cc:1035] T 74b7ff29e276451985e7cbe72ceff148 P 6d231fa7f1684535af46a9b196111880 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:45:43.442541 1024 raft_consensus.cc:1261] T a6c9b839b6a346aca42a280aed623913 P ed5783865eb74d39bee2a38e9139dea4 [term 2 FOLLOWER]: Refusing update from remote peer f7fd7bf659f84ae184696b891db5221a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 20:45:43.443017 2891 consensus_queue.cc:1035] T a6c9b839b6a346aca42a280aed623913 P f7fd7bf659f84ae184696b891db5221a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:45:43.450275 2157 consensus_queue.cc:237] T 21a27f13d4104a8a87aa8b935c6c8dfa P 2260afde4ccb498189d7b25c00a2c727 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2260afde4ccb498189d7b25c00a2c727" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 40723 } attrs { promote: false } }
I20240502 20:45:43.450874 3294 raft_consensus.cc:2932] T 21a27f13d4104a8a87aa8b935c6c8dfa P 2260afde4ccb498189d7b25c00a2c727 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 33be9a5b29214a8d84c2fd7a4936a920 (127.30.32.65) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2260afde4ccb498189d7b25c00a2c727" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 40723 } attrs { promote: false } } }
I20240502 20:45:43.452131 735 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 21a27f13d4104a8a87aa8b935c6c8dfa with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 20:45:43.453060 744 catalog_manager.cc:5484] T 21a27f13d4104a8a87aa8b935c6c8dfa P 2260afde4ccb498189d7b25c00a2c727 reported cstate change: config changed from index 5 to 7, VOTER 33be9a5b29214a8d84c2fd7a4936a920 (127.30.32.65) evicted. New cstate: current_term: 2 leader_uuid: "2260afde4ccb498189d7b25c00a2c727" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2260afde4ccb498189d7b25c00a2c727" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 40723 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:43.455315 873 tablet_service.cc:1514] Processing DeleteTablet for tablet 21a27f13d4104a8a87aa8b935c6c8dfa with delete_type TABLET_DATA_TOMBSTONED (TS 33be9a5b29214a8d84c2fd7a4936a920 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:54242
I20240502 20:45:43.455538 3353 tablet_replica.cc:330] T 21a27f13d4104a8a87aa8b935c6c8dfa P 33be9a5b29214a8d84c2fd7a4936a920: stopping tablet replica
I20240502 20:45:43.455649 3353 raft_consensus.cc:2229] T 21a27f13d4104a8a87aa8b935c6c8dfa P 33be9a5b29214a8d84c2fd7a4936a920 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:43.455740 3353 raft_consensus.cc:2258] T 21a27f13d4104a8a87aa8b935c6c8dfa P 33be9a5b29214a8d84c2fd7a4936a920 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:43.456348 3353 ts_tablet_manager.cc:1875] T 21a27f13d4104a8a87aa8b935c6c8dfa P 33be9a5b29214a8d84c2fd7a4936a920: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:45:43.458263 3353 ts_tablet_manager.cc:1888] T 21a27f13d4104a8a87aa8b935c6c8dfa P 33be9a5b29214a8d84c2fd7a4936a920: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 20:45:43.458331 3353 log.cc:1196] T 21a27f13d4104a8a87aa8b935c6c8dfa P 33be9a5b29214a8d84c2fd7a4936a920: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal/wals/21a27f13d4104a8a87aa8b935c6c8dfa
I20240502 20:45:43.458884 735 catalog_manager.cc:4830] TS 33be9a5b29214a8d84c2fd7a4936a920 (127.30.32.65:45537): tablet 21a27f13d4104a8a87aa8b935c6c8dfa (table rebalance_test_table_2 [id=fddf24b281f54ee394d7a819baadce47]) successfully deleted
I20240502 20:45:43.465735 2025 consensus_queue.cc:237] T da05217d82a74bd6aeb54330c3d6d370 P f7fd7bf659f84ae184696b891db5221a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f7fd7bf659f84ae184696b891db5221a" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42407 } attrs { promote: false } }
I20240502 20:45:43.466405 3297 raft_consensus.cc:2932] T da05217d82a74bd6aeb54330c3d6d370 P f7fd7bf659f84ae184696b891db5221a [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 33be9a5b29214a8d84c2fd7a4936a920 (127.30.32.65) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f7fd7bf659f84ae184696b891db5221a" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42407 } attrs { promote: false } } }
I20240502 20:45:43.467674 738 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet da05217d82a74bd6aeb54330c3d6d370 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 20:45:43.468561 744 catalog_manager.cc:5484] T da05217d82a74bd6aeb54330c3d6d370 P f7fd7bf659f84ae184696b891db5221a reported cstate change: config changed from index 5 to 7, VOTER 33be9a5b29214a8d84c2fd7a4936a920 (127.30.32.65) evicted. New cstate: current_term: 2 leader_uuid: "f7fd7bf659f84ae184696b891db5221a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f7fd7bf659f84ae184696b891db5221a" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42407 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:43.471102 873 tablet_service.cc:1514] Processing DeleteTablet for tablet da05217d82a74bd6aeb54330c3d6d370 with delete_type TABLET_DATA_TOMBSTONED (TS 33be9a5b29214a8d84c2fd7a4936a920 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:54242
I20240502 20:45:43.471372 3353 tablet_replica.cc:330] T da05217d82a74bd6aeb54330c3d6d370 P 33be9a5b29214a8d84c2fd7a4936a920: stopping tablet replica
I20240502 20:45:43.471518 3353 raft_consensus.cc:2229] T da05217d82a74bd6aeb54330c3d6d370 P 33be9a5b29214a8d84c2fd7a4936a920 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:43.471623 3353 raft_consensus.cc:2258] T da05217d82a74bd6aeb54330c3d6d370 P 33be9a5b29214a8d84c2fd7a4936a920 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:43.472170 3353 ts_tablet_manager.cc:1875] T da05217d82a74bd6aeb54330c3d6d370 P 33be9a5b29214a8d84c2fd7a4936a920: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:45:43.473953 3353 ts_tablet_manager.cc:1888] T da05217d82a74bd6aeb54330c3d6d370 P 33be9a5b29214a8d84c2fd7a4936a920: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 20:45:43.474021 3353 log.cc:1196] T da05217d82a74bd6aeb54330c3d6d370 P 33be9a5b29214a8d84c2fd7a4936a920: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal/wals/da05217d82a74bd6aeb54330c3d6d370
I20240502 20:45:43.474520 735 catalog_manager.cc:4830] TS 33be9a5b29214a8d84c2fd7a4936a920 (127.30.32.65:45537): tablet da05217d82a74bd6aeb54330c3d6d370 (table rebalance_test_table_3 [id=2a2a74b349514a8aa4c6c02c2533a51f]) successfully deleted
I20240502 20:45:43.535208 2025 consensus_queue.cc:237] T a6c9b839b6a346aca42a280aed623913 P f7fd7bf659f84ae184696b891db5221a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f7fd7bf659f84ae184696b891db5221a" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42407 } attrs { promote: false } }
I20240502 20:45:43.535914 3297 raft_consensus.cc:2932] T a6c9b839b6a346aca42a280aed623913 P f7fd7bf659f84ae184696b891db5221a [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER ed5783865eb74d39bee2a38e9139dea4 (127.30.32.66) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f7fd7bf659f84ae184696b891db5221a" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42407 } attrs { promote: false } } }
I20240502 20:45:43.537181 738 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a6c9b839b6a346aca42a280aed623913 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 20:45:43.538113 744 catalog_manager.cc:5484] T a6c9b839b6a346aca42a280aed623913 P f7fd7bf659f84ae184696b891db5221a reported cstate change: config changed from index 5 to 7, VOTER ed5783865eb74d39bee2a38e9139dea4 (127.30.32.66) evicted. New cstate: current_term: 2 leader_uuid: "f7fd7bf659f84ae184696b891db5221a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f7fd7bf659f84ae184696b891db5221a" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42407 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:43.540630 1003 tablet_service.cc:1514] Processing DeleteTablet for tablet a6c9b839b6a346aca42a280aed623913 with delete_type TABLET_DATA_TOMBSTONED (TS ed5783865eb74d39bee2a38e9139dea4 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57638
I20240502 20:45:43.541009 3352 tablet_replica.cc:330] T a6c9b839b6a346aca42a280aed623913 P ed5783865eb74d39bee2a38e9139dea4: stopping tablet replica
I20240502 20:45:43.541168 3352 raft_consensus.cc:2229] T a6c9b839b6a346aca42a280aed623913 P ed5783865eb74d39bee2a38e9139dea4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:43.541296 3352 raft_consensus.cc:2258] T a6c9b839b6a346aca42a280aed623913 P ed5783865eb74d39bee2a38e9139dea4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:43.541345 2428 consensus_queue.cc:237] T 74b7ff29e276451985e7cbe72ceff148 P 6d231fa7f1684535af46a9b196111880 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d231fa7f1684535af46a9b196111880" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 43291 } attrs { promote: false } }
I20240502 20:45:43.541997 3352 ts_tablet_manager.cc:1875] T a6c9b839b6a346aca42a280aed623913 P ed5783865eb74d39bee2a38e9139dea4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:45:43.542102 3299 raft_consensus.cc:2932] T 74b7ff29e276451985e7cbe72ceff148 P 6d231fa7f1684535af46a9b196111880 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER ed5783865eb74d39bee2a38e9139dea4 (127.30.32.66) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d231fa7f1684535af46a9b196111880" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 43291 } attrs { promote: false } } }
I20240502 20:45:43.543493 736 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 74b7ff29e276451985e7cbe72ceff148 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 20:45:43.544312 744 catalog_manager.cc:5484] T 74b7ff29e276451985e7cbe72ceff148 P 6d231fa7f1684535af46a9b196111880 reported cstate change: config changed from index 5 to 7, VOTER ed5783865eb74d39bee2a38e9139dea4 (127.30.32.66) evicted. New cstate: current_term: 2 leader_uuid: "6d231fa7f1684535af46a9b196111880" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d231fa7f1684535af46a9b196111880" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 43291 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:43.544503 3352 ts_tablet_manager.cc:1888] T a6c9b839b6a346aca42a280aed623913 P ed5783865eb74d39bee2a38e9139dea4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 20:45:43.544575 3352 log.cc:1196] T a6c9b839b6a346aca42a280aed623913 P ed5783865eb74d39bee2a38e9139dea4: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal/wals/a6c9b839b6a346aca42a280aed623913
I20240502 20:45:43.545161 738 catalog_manager.cc:4830] TS ed5783865eb74d39bee2a38e9139dea4 (127.30.32.66:45113): tablet a6c9b839b6a346aca42a280aed623913 (table rebalance_test_table_1 [id=da321cf3a7944f5ba716ef84b816582e]) successfully deleted
I20240502 20:45:43.546999 1003 tablet_service.cc:1514] Processing DeleteTablet for tablet 74b7ff29e276451985e7cbe72ceff148 with delete_type TABLET_DATA_TOMBSTONED (TS ed5783865eb74d39bee2a38e9139dea4 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57638
I20240502 20:45:43.547638 3352 tablet_replica.cc:330] T 74b7ff29e276451985e7cbe72ceff148 P ed5783865eb74d39bee2a38e9139dea4: stopping tablet replica
I20240502 20:45:43.547801 3352 raft_consensus.cc:2229] T 74b7ff29e276451985e7cbe72ceff148 P ed5783865eb74d39bee2a38e9139dea4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:43.547989 3352 raft_consensus.cc:2258] T 74b7ff29e276451985e7cbe72ceff148 P ed5783865eb74d39bee2a38e9139dea4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:43.548579 3352 ts_tablet_manager.cc:1875] T 74b7ff29e276451985e7cbe72ceff148 P ed5783865eb74d39bee2a38e9139dea4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:45:43.550153 3352 ts_tablet_manager.cc:1888] T 74b7ff29e276451985e7cbe72ceff148 P ed5783865eb74d39bee2a38e9139dea4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 20:45:43.550230 3352 log.cc:1196] T 74b7ff29e276451985e7cbe72ceff148 P ed5783865eb74d39bee2a38e9139dea4: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal/wals/74b7ff29e276451985e7cbe72ceff148
I20240502 20:45:43.550823 738 catalog_manager.cc:4830] TS ed5783865eb74d39bee2a38e9139dea4 (127.30.32.66:45113): tablet 74b7ff29e276451985e7cbe72ceff148 (table rebalance_test_table_0 [id=9738245a8d55447f94af91ee9ca2ce96]) successfully deleted
I20240502 20:45:43.630170 2137 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:43.632539 2539 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:43.633251 2277 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:43.641295 873 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:45:43.644124 2408 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:43.652036 2005 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:43.655414 1003 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:45:43.742624 2539 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:43.749181 873 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:45:43.749208 2137 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:43.750523 2408 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:43.751689 2277 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:43.758873 1003 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:45:43.767158 2005 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:43.987735 2539 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:43.990561 2137 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:43.991206 873 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:45:43.991953 1003 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:45:44.001353 2277 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:44.007282 2408 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:44.025751 2005 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:44.134912 2277 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:44.135372 2539 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:44.136791 2408 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:44.139182 1003 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:45:44.147933 873 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:45:44.159291 2137 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:44.168638 2005 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:44.265182 2277 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:44.266350 2408 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:44.266638 2539 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:44.270112 2137 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:44.280633 873 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:45:44.280730 2005 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:44.294121 1003 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
5c223b22641f4581b6ec8c5e553014cd | 127.30.32.126:43021 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.30.32.84:35991 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
0dfc68b63d15462398c76bfbff0e3cf8 | 127.30.32.71:40439 | HEALTHY | <none> | 2 | 0
2260afde4ccb498189d7b25c00a2c727 | 127.30.32.68:40723 | HEALTHY | <none> | 2 | 0
33be9a5b29214a8d84c2fd7a4936a920 | 127.30.32.65:45537 | HEALTHY | <none> | 3 | 0
6d231fa7f1684535af46a9b196111880 | 127.30.32.70:43291 | HEALTHY | <none> | 2 | 0
cec988e98db64e2f98b030efc16d4108 | 127.30.32.69:34789 | HEALTHY | <none> | 2 | 0
ed5783865eb74d39bee2a38e9139dea4 | 127.30.32.66:45113 | HEALTHY | <none> | 3 | 0
f7fd7bf659f84ae184696b891db5221a | 127.30.32.67:42407 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.30.32.65 | experimental | 127.30.32.65:45537
local_ip_for_outbound_sockets | 127.30.32.66 | experimental | 127.30.32.66:45113
local_ip_for_outbound_sockets | 127.30.32.67 | experimental | 127.30.32.67:42407
local_ip_for_outbound_sockets | 127.30.32.68 | experimental | 127.30.32.68:40723
local_ip_for_outbound_sockets | 127.30.32.69 | experimental | 127.30.32.69:34789
local_ip_for_outbound_sockets | 127.30.32.70 | experimental | 127.30.32.70:43291
local_ip_for_outbound_sockets | 127.30.32.71 | experimental | 127.30.32.71:40439
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.30.32.65:45537
server_dump_info_path | /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.30.32.66:45113
server_dump_info_path | /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.30.32.67:42407
server_dump_info_path | /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.30.32.68:40723
server_dump_info_path | /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.30.32.69:34789
server_dump_info_path | /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.30.32.70:43291
server_dump_info_path | /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.30.32.71:40439
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.30.32.84:35991 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 20:45:44.319499 30849 log_verifier.cc:126] Checking tablet 0734f8b2eeb04ad99ff827c0ecf90663
I20240502 20:45:44.331152 30849 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 0734f8b2eeb04ad99ff827c0ecf90663
I20240502 20:45:44.331252 30849 log_verifier.cc:126] Checking tablet 21a27f13d4104a8a87aa8b935c6c8dfa
I20240502 20:45:44.340090 30849 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 21a27f13d4104a8a87aa8b935c6c8dfa
I20240502 20:45:44.340165 30849 log_verifier.cc:126] Checking tablet 49eb89bf68d942398c68e2bf1d0f6370
I20240502 20:45:44.349014 30849 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 49eb89bf68d942398c68e2bf1d0f6370
I20240502 20:45:44.349090 30849 log_verifier.cc:126] Checking tablet 63e2878c2981429cbcb41837f3f0c44a
I20240502 20:45:44.357529 30849 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 63e2878c2981429cbcb41837f3f0c44a
I20240502 20:45:44.357595 30849 log_verifier.cc:126] Checking tablet 74b7ff29e276451985e7cbe72ceff148
I20240502 20:45:44.366143 30849 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 74b7ff29e276451985e7cbe72ceff148
I20240502 20:45:44.366205 30849 log_verifier.cc:126] Checking tablet 7a888b6096c2427f8a7f67b26c3c00b7
I20240502 20:45:44.374984 30849 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 7a888b6096c2427f8a7f67b26c3c00b7
I20240502 20:45:44.375054 30849 log_verifier.cc:126] Checking tablet 83593d702ac542c28bd1721338a55c5c
I20240502 20:45:44.384263 30849 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 83593d702ac542c28bd1721338a55c5c
I20240502 20:45:44.384347 30849 log_verifier.cc:126] Checking tablet 89fc553dd07e4af5bd37801a34c4ac66
I20240502 20:45:44.393255 30849 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 89fc553dd07e4af5bd37801a34c4ac66
I20240502 20:45:44.393321 30849 log_verifier.cc:126] Checking tablet 8d06cc46e1b04a809b88c070996ba1ed
I20240502 20:45:44.401834 30849 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 8d06cc46e1b04a809b88c070996ba1ed
I20240502 20:45:44.401901 30849 log_verifier.cc:126] Checking tablet 975b1d549985459da90b6cde03b0c5b8
I20240502 20:45:44.410519 30849 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 975b1d549985459da90b6cde03b0c5b8
I20240502 20:45:44.410588 30849 log_verifier.cc:126] Checking tablet a6c9b839b6a346aca42a280aed623913
I20240502 20:45:44.419490 30849 log_verifier.cc:177] Verified matching terms for 7 ops in tablet a6c9b839b6a346aca42a280aed623913
I20240502 20:45:44.419565 30849 log_verifier.cc:126] Checking tablet afc220634a164930bdaa8ece818fb2e7
I20240502 20:45:44.428326 30849 log_verifier.cc:177] Verified matching terms for 7 ops in tablet afc220634a164930bdaa8ece818fb2e7
I20240502 20:45:44.428393 30849 log_verifier.cc:126] Checking tablet cf7de28bcb20418f99846c121cd88531
I20240502 20:45:44.437352 30849 log_verifier.cc:177] Verified matching terms for 7 ops in tablet cf7de28bcb20418f99846c121cd88531
I20240502 20:45:44.437419 30849 log_verifier.cc:126] Checking tablet da05217d82a74bd6aeb54330c3d6d370
I20240502 20:45:44.446287 30849 log_verifier.cc:177] Verified matching terms for 7 ops in tablet da05217d82a74bd6aeb54330c3d6d370
I20240502 20:45:44.446362 30849 log_verifier.cc:126] Checking tablet e7080f04f87d4a6f926edea263b450bc
I20240502 20:45:44.454358 30849 log_verifier.cc:177] Verified matching terms for 2 ops in tablet e7080f04f87d4a6f926edea263b450bc
I20240502 20:45:44.454428 30849 log_verifier.cc:126] Checking tablet ff0d71af3fd4483caa41325903f9ac39
I20240502 20:45:44.463272 30849 log_verifier.cc:177] Verified matching terms for 3 ops in tablet ff0d71af3fd4483caa41325903f9ac39
I20240502 20:45:44.465454 30849 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskolF7xA/build/debug/bin/kudu
/tmp/dist-test-taskolF7xA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.32.72:0
--local_ip_for_outbound_sockets=127.30.32.72
--webserver_interface=127.30.32.72
--webserver_port=0
--tserver_master_addrs=127.30.32.126:43021
--builtin_ntp_servers=127.30.32.84:35991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 20:45:44.574453 3658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:44.574676 3658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:44.574728 3658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:44.577944 3658 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:44.578015 3658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:44.578112 3658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.32.72
I20240502 20:45:44.582024 3658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.32.84:35991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.32.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.30.32.72
--webserver_port=0
--tserver_master_addrs=127.30.32.126:43021
--never_fsync=true
--heap_profile_path=/tmp/kudu.3658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.32.72
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:44.582983 3658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:44.584157 3658 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:44.593113 3664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:44.593551 3663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:44.594273 3666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:44.594987 3658 server_base.cc:1034] running on GCE node
I20240502 20:45:44.595413 3658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:44.595981 3658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:44.597141 3658 hybrid_clock.cc:648] HybridClock initialized: now 1714682744597127 us; error 27 us; skew 500 ppm
I20240502 20:45:44.599056 3658 webserver.cc:460] Webserver started at http://127.30.32.72:46571/ using document root <none> and password file <none>
I20240502 20:45:44.599622 3658 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:44.599680 3658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:44.599860 3658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:44.601649 3658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "d740107bba7b4fa290be7e085ddd55c6"
format_stamp: "Formatted at 2024-05-02 20:45:44 on dist-test-slave-6336"
I20240502 20:45:44.602125 3658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "d740107bba7b4fa290be7e085ddd55c6"
format_stamp: "Formatted at 2024-05-02 20:45:44 on dist-test-slave-6336"
I20240502 20:45:44.605443 3658 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 20:45:44.607646 3672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:44.608620 3658 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:44.608726 3658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "d740107bba7b4fa290be7e085ddd55c6"
format_stamp: "Formatted at 2024-05-02 20:45:44 on dist-test-slave-6336"
I20240502 20:45:44.608817 3658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:44.618922 3658 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:44.619652 3658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:44.619802 3658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:44.620402 3658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:44.621362 3658 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:44.621410 3658 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:44.621455 3658 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:44.621484 3658 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:44.630141 3658 rpc_server.cc:307] RPC server started. Bound to: 127.30.32.72:38655
I20240502 20:45:44.630162 3785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.32.72:38655 every 8 connection(s)
I20240502 20:45:44.631213 3658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 20:45:44.631397 30849 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskolF7xA/build/debug/bin/kudu as pid 3658
I20240502 20:45:44.631507 30849 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 20:45:44.642164 3786 heartbeater.cc:347] Connected to a master server at 127.30.32.126:43021
I20240502 20:45:44.642426 3786 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:44.643000 3786 heartbeater.cc:511] Master 127.30.32.126:43021 requested a full tablet report, sending...
I20240502 20:45:44.644002 751 ts_manager.cc:194] Registered new tserver with Master: d740107bba7b4fa290be7e085ddd55c6 (127.30.32.72:38655)
I20240502 20:45:44.644593 751 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.32.72:43053
I20240502 20:45:44.833803 1003 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:45:44.837642 2137 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:44.837932 2277 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:44.841138 2408 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:44.845144 2539 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:44.859617 2005 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:44.868749 3720 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:44.870726 873 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:45:44.987306 2408 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:44.992837 873 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:45:44.995014 2277 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:44.995155 2539 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:45.006304 3720 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:45.013211 1003 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:45:45.014109 2137 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:45.019428 2005 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:45.057296 1024 consensus_queue.cc:237] T 8d06cc46e1b04a809b88c070996ba1ed P ed5783865eb74d39bee2a38e9139dea4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } } peers { permanent_uuid: "d740107bba7b4fa290be7e085ddd55c6" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 38655 } attrs { promote: true } }
I20240502 20:45:45.058269 3879 raft_consensus.cc:2932] T 8d06cc46e1b04a809b88c070996ba1ed P ed5783865eb74d39bee2a38e9139dea4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d740107bba7b4fa290be7e085ddd55c6 (127.30.32.72) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } } peers { permanent_uuid: "d740107bba7b4fa290be7e085ddd55c6" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 38655 } attrs { promote: true } } }
I20240502 20:45:45.063007 746 catalog_manager.cc:5484] T 8d06cc46e1b04a809b88c070996ba1ed P ed5783865eb74d39bee2a38e9139dea4 reported cstate change: config changed from index -1 to 4, NON_VOTER d740107bba7b4fa290be7e085ddd55c6 (127.30.32.72) added. New cstate: current_term: 1 leader_uuid: "ed5783865eb74d39bee2a38e9139dea4" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d740107bba7b4fa290be7e085ddd55c6" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 38655 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 20:45:45.064143 960 consensus_peers.cc:485] T 8d06cc46e1b04a809b88c070996ba1ed P ed5783865eb74d39bee2a38e9139dea4 -> Peer d740107bba7b4fa290be7e085ddd55c6 (127.30.32.72:38655): Couldn't send request to peer d740107bba7b4fa290be7e085ddd55c6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8d06cc46e1b04a809b88c070996ba1ed. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:45.070494 893 consensus_queue.cc:237] T 63e2878c2981429cbcb41837f3f0c44a P 33be9a5b29214a8d84c2fd7a4936a920 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "d740107bba7b4fa290be7e085ddd55c6" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 38655 } attrs { promote: true } }
I20240502 20:45:45.071432 3892 raft_consensus.cc:2932] T 63e2878c2981429cbcb41837f3f0c44a P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d740107bba7b4fa290be7e085ddd55c6 (127.30.32.72) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "d740107bba7b4fa290be7e085ddd55c6" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 38655 } attrs { promote: true } } }
I20240502 20:45:45.075111 746 catalog_manager.cc:5484] T 63e2878c2981429cbcb41837f3f0c44a P 33be9a5b29214a8d84c2fd7a4936a920 reported cstate change: config changed from index -1 to 4, NON_VOTER d740107bba7b4fa290be7e085ddd55c6 (127.30.32.72) added. New cstate: current_term: 1 leader_uuid: "33be9a5b29214a8d84c2fd7a4936a920" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d740107bba7b4fa290be7e085ddd55c6" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 38655 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 20:45:45.075557 829 consensus_peers.cc:485] T 63e2878c2981429cbcb41837f3f0c44a P 33be9a5b29214a8d84c2fd7a4936a920 -> Peer d740107bba7b4fa290be7e085ddd55c6 (127.30.32.72:38655): Couldn't send request to peer d740107bba7b4fa290be7e085ddd55c6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63e2878c2981429cbcb41837f3f0c44a. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:45.438151 3915 ts_tablet_manager.cc:902] T 8d06cc46e1b04a809b88c070996ba1ed P d740107bba7b4fa290be7e085ddd55c6: Initiating tablet copy from peer ed5783865eb74d39bee2a38e9139dea4 (127.30.32.66:45113)
I20240502 20:45:45.438881 3915 tablet_copy_client.cc:308] T 8d06cc46e1b04a809b88c070996ba1ed P d740107bba7b4fa290be7e085ddd55c6: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.66:45113
I20240502 20:45:45.442394 1044 tablet_copy_service.cc:140] P ed5783865eb74d39bee2a38e9139dea4: Received BeginTabletCopySession request for tablet 8d06cc46e1b04a809b88c070996ba1ed from peer d740107bba7b4fa290be7e085ddd55c6 ({username='slave'} at 127.30.32.72:52787)
I20240502 20:45:45.442492 1044 tablet_copy_service.cc:161] P ed5783865eb74d39bee2a38e9139dea4: Beginning new tablet copy session on tablet 8d06cc46e1b04a809b88c070996ba1ed from peer d740107bba7b4fa290be7e085ddd55c6 at {username='slave'} at 127.30.32.72:52787: session id = d740107bba7b4fa290be7e085ddd55c6-8d06cc46e1b04a809b88c070996ba1ed
I20240502 20:45:45.443233 1044 tablet_copy_source_session.cc:205] T 8d06cc46e1b04a809b88c070996ba1ed P ed5783865eb74d39bee2a38e9139dea4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:45:45.446281 3915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d06cc46e1b04a809b88c070996ba1ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:45.451004 3915 tablet_copy_client.cc:783] T 8d06cc46e1b04a809b88c070996ba1ed P d740107bba7b4fa290be7e085ddd55c6: tablet copy: Starting download of 0 data blocks...
I20240502 20:45:45.451189 3915 tablet_copy_client.cc:647] T 8d06cc46e1b04a809b88c070996ba1ed P d740107bba7b4fa290be7e085ddd55c6: tablet copy: Starting download of 1 WAL segments...
I20240502 20:45:45.452615 3915 tablet_copy_client.cc:523] T 8d06cc46e1b04a809b88c070996ba1ed P d740107bba7b4fa290be7e085ddd55c6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:45:45.454150 3915 tablet_bootstrap.cc:492] T 8d06cc46e1b04a809b88c070996ba1ed P d740107bba7b4fa290be7e085ddd55c6: Bootstrap starting.
I20240502 20:45:45.459564 3915 log.cc:824] T 8d06cc46e1b04a809b88c070996ba1ed P d740107bba7b4fa290be7e085ddd55c6: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:45.479136 3915 tablet_bootstrap.cc:492] T 8d06cc46e1b04a809b88c070996ba1ed P d740107bba7b4fa290be7e085ddd55c6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:45:45.479496 3915 tablet_bootstrap.cc:492] T 8d06cc46e1b04a809b88c070996ba1ed P d740107bba7b4fa290be7e085ddd55c6: Bootstrap complete.
I20240502 20:45:45.479797 3915 ts_tablet_manager.cc:1367] T 8d06cc46e1b04a809b88c070996ba1ed P d740107bba7b4fa290be7e085ddd55c6: Time spent bootstrapping tablet: real 0.026s user 0.023s sys 0.001s
I20240502 20:45:45.481788 3915 raft_consensus.cc:348] T 8d06cc46e1b04a809b88c070996ba1ed P d740107bba7b4fa290be7e085ddd55c6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } } peers { permanent_uuid: "d740107bba7b4fa290be7e085ddd55c6" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 38655 } attrs { promote: true } }
I20240502 20:45:45.481994 3915 raft_consensus.cc:729] T 8d06cc46e1b04a809b88c070996ba1ed P d740107bba7b4fa290be7e085ddd55c6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d740107bba7b4fa290be7e085ddd55c6, State: Initialized, Role: LEARNER
I20240502 20:45:45.482298 3915 consensus_queue.cc:260] T 8d06cc46e1b04a809b88c070996ba1ed P d740107bba7b4fa290be7e085ddd55c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } } peers { permanent_uuid: "d740107bba7b4fa290be7e085ddd55c6" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 38655 } attrs { promote: true } }
I20240502 20:45:45.482981 3786 heartbeater.cc:503] Master 127.30.32.126:43021 was elected leader, sending a full tablet report...
I20240502 20:45:45.483117 3915 ts_tablet_manager.cc:1398] T 8d06cc46e1b04a809b88c070996ba1ed P d740107bba7b4fa290be7e085ddd55c6: Time spent starting tablet: real 0.003s user 0.006s sys 0.000s
I20240502 20:45:45.483856 1044 tablet_copy_service.cc:342] P ed5783865eb74d39bee2a38e9139dea4: Request end of tablet copy session d740107bba7b4fa290be7e085ddd55c6-8d06cc46e1b04a809b88c070996ba1ed received from {username='slave'} at 127.30.32.72:52787
I20240502 20:45:45.483960 1044 tablet_copy_service.cc:433] P ed5783865eb74d39bee2a38e9139dea4: ending tablet copy session d740107bba7b4fa290be7e085ddd55c6-8d06cc46e1b04a809b88c070996ba1ed on tablet 8d06cc46e1b04a809b88c070996ba1ed with peer d740107bba7b4fa290be7e085ddd55c6
I20240502 20:45:45.528766 3915 ts_tablet_manager.cc:902] T 63e2878c2981429cbcb41837f3f0c44a P d740107bba7b4fa290be7e085ddd55c6: Initiating tablet copy from peer 33be9a5b29214a8d84c2fd7a4936a920 (127.30.32.65:45537)
I20240502 20:45:45.529304 3915 tablet_copy_client.cc:308] T 63e2878c2981429cbcb41837f3f0c44a P d740107bba7b4fa290be7e085ddd55c6: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.65:45537
I20240502 20:45:45.532526 905 tablet_copy_service.cc:140] P 33be9a5b29214a8d84c2fd7a4936a920: Received BeginTabletCopySession request for tablet 63e2878c2981429cbcb41837f3f0c44a from peer d740107bba7b4fa290be7e085ddd55c6 ({username='slave'} at 127.30.32.72:53557)
I20240502 20:45:45.532660 905 tablet_copy_service.cc:161] P 33be9a5b29214a8d84c2fd7a4936a920: Beginning new tablet copy session on tablet 63e2878c2981429cbcb41837f3f0c44a from peer d740107bba7b4fa290be7e085ddd55c6 at {username='slave'} at 127.30.32.72:53557: session id = d740107bba7b4fa290be7e085ddd55c6-63e2878c2981429cbcb41837f3f0c44a
I20240502 20:45:45.533478 905 tablet_copy_source_session.cc:205] T 63e2878c2981429cbcb41837f3f0c44a P 33be9a5b29214a8d84c2fd7a4936a920: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:45:45.534405 3915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63e2878c2981429cbcb41837f3f0c44a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:45.536207 3915 tablet_copy_client.cc:783] T 63e2878c2981429cbcb41837f3f0c44a P d740107bba7b4fa290be7e085ddd55c6: tablet copy: Starting download of 0 data blocks...
I20240502 20:45:45.536377 3915 tablet_copy_client.cc:647] T 63e2878c2981429cbcb41837f3f0c44a P d740107bba7b4fa290be7e085ddd55c6: tablet copy: Starting download of 1 WAL segments...
I20240502 20:45:45.537940 3915 tablet_copy_client.cc:523] T 63e2878c2981429cbcb41837f3f0c44a P d740107bba7b4fa290be7e085ddd55c6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:45:45.539005 3915 tablet_bootstrap.cc:492] T 63e2878c2981429cbcb41837f3f0c44a P d740107bba7b4fa290be7e085ddd55c6: Bootstrap starting.
I20240502 20:45:45.556298 3915 tablet_bootstrap.cc:492] T 63e2878c2981429cbcb41837f3f0c44a P d740107bba7b4fa290be7e085ddd55c6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:45:45.556589 3915 tablet_bootstrap.cc:492] T 63e2878c2981429cbcb41837f3f0c44a P d740107bba7b4fa290be7e085ddd55c6: Bootstrap complete.
I20240502 20:45:45.556766 3915 ts_tablet_manager.cc:1367] T 63e2878c2981429cbcb41837f3f0c44a P d740107bba7b4fa290be7e085ddd55c6: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.000s
I20240502 20:45:45.557358 3915 raft_consensus.cc:348] T 63e2878c2981429cbcb41837f3f0c44a P d740107bba7b4fa290be7e085ddd55c6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "d740107bba7b4fa290be7e085ddd55c6" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 38655 } attrs { promote: true } }
I20240502 20:45:45.557471 3915 raft_consensus.cc:729] T 63e2878c2981429cbcb41837f3f0c44a P d740107bba7b4fa290be7e085ddd55c6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d740107bba7b4fa290be7e085ddd55c6, State: Initialized, Role: LEARNER
I20240502 20:45:45.557575 3915 consensus_queue.cc:260] T 63e2878c2981429cbcb41837f3f0c44a P d740107bba7b4fa290be7e085ddd55c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "d740107bba7b4fa290be7e085ddd55c6" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 38655 } attrs { promote: true } }
I20240502 20:45:45.557834 3915 ts_tablet_manager.cc:1398] T 63e2878c2981429cbcb41837f3f0c44a P d740107bba7b4fa290be7e085ddd55c6: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240502 20:45:45.558496 905 tablet_copy_service.cc:342] P 33be9a5b29214a8d84c2fd7a4936a920: Request end of tablet copy session d740107bba7b4fa290be7e085ddd55c6-63e2878c2981429cbcb41837f3f0c44a received from {username='slave'} at 127.30.32.72:53557
I20240502 20:45:45.558579 905 tablet_copy_service.cc:433] P 33be9a5b29214a8d84c2fd7a4936a920: ending tablet copy session d740107bba7b4fa290be7e085ddd55c6-63e2878c2981429cbcb41837f3f0c44a on tablet 63e2878c2981429cbcb41837f3f0c44a with peer d740107bba7b4fa290be7e085ddd55c6
I20240502 20:45:45.817327 3740 raft_consensus.cc:1203] T 8d06cc46e1b04a809b88c070996ba1ed P d740107bba7b4fa290be7e085ddd55c6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:45:46.110443 3740 raft_consensus.cc:1203] T 63e2878c2981429cbcb41837f3f0c44a P d740107bba7b4fa290be7e085ddd55c6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:45:46.236804 3879 raft_consensus.cc:1050] T 8d06cc46e1b04a809b88c070996ba1ed P ed5783865eb74d39bee2a38e9139dea4: attempting to promote NON_VOTER d740107bba7b4fa290be7e085ddd55c6 to VOTER
I20240502 20:45:46.237174 3879 consensus_queue.cc:237] T 8d06cc46e1b04a809b88c070996ba1ed P ed5783865eb74d39bee2a38e9139dea4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } } peers { permanent_uuid: "d740107bba7b4fa290be7e085ddd55c6" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 38655 } attrs { promote: false } }
I20240502 20:45:46.238292 3740 raft_consensus.cc:1261] T 8d06cc46e1b04a809b88c070996ba1ed P d740107bba7b4fa290be7e085ddd55c6 [term 1 LEARNER]: Refusing update from remote peer ed5783865eb74d39bee2a38e9139dea4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:45:46.238716 3879 consensus_queue.cc:1035] T 8d06cc46e1b04a809b88c070996ba1ed P ed5783865eb74d39bee2a38e9139dea4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d740107bba7b4fa290be7e085ddd55c6" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 38655 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:45:46.241351 3949 raft_consensus.cc:2932] T 8d06cc46e1b04a809b88c070996ba1ed P ed5783865eb74d39bee2a38e9139dea4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d740107bba7b4fa290be7e085ddd55c6 (127.30.32.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } } peers { permanent_uuid: "d740107bba7b4fa290be7e085ddd55c6" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 38655 } attrs { promote: false } } }
I20240502 20:45:46.241848 3740 raft_consensus.cc:2932] T 8d06cc46e1b04a809b88c070996ba1ed P d740107bba7b4fa290be7e085ddd55c6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d740107bba7b4fa290be7e085ddd55c6 (127.30.32.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } } peers { permanent_uuid: "d740107bba7b4fa290be7e085ddd55c6" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 38655 } attrs { promote: false } } }
I20240502 20:45:46.243891 743 catalog_manager.cc:5484] T 8d06cc46e1b04a809b88c070996ba1ed P ed5783865eb74d39bee2a38e9139dea4 reported cstate change: config changed from index 4 to 5, d740107bba7b4fa290be7e085ddd55c6 (127.30.32.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ed5783865eb74d39bee2a38e9139dea4" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d740107bba7b4fa290be7e085ddd55c6" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 38655 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:46.368474 1024 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8d06cc46e1b04a809b88c070996ba1ed"
dest_uuid: "ed5783865eb74d39bee2a38e9139dea4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59150
I20240502 20:45:46.368610 1024 raft_consensus.cc:595] T 8d06cc46e1b04a809b88c070996ba1ed P ed5783865eb74d39bee2a38e9139dea4 [term 1 LEADER]: Received request to transfer leadership
I20240502 20:45:46.585925 1024 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8d06cc46e1b04a809b88c070996ba1ed"
dest_uuid: "ed5783865eb74d39bee2a38e9139dea4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59166
I20240502 20:45:46.586097 1024 raft_consensus.cc:595] T 8d06cc46e1b04a809b88c070996ba1ed P ed5783865eb74d39bee2a38e9139dea4 [term 1 LEADER]: Received request to transfer leadership
I20240502 20:45:46.662889 4009 raft_consensus.cc:1050] T 63e2878c2981429cbcb41837f3f0c44a P 33be9a5b29214a8d84c2fd7a4936a920: attempting to promote NON_VOTER d740107bba7b4fa290be7e085ddd55c6 to VOTER
I20240502 20:45:46.663282 4009 consensus_queue.cc:237] T 63e2878c2981429cbcb41837f3f0c44a P 33be9a5b29214a8d84c2fd7a4936a920 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "d740107bba7b4fa290be7e085ddd55c6" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 38655 } attrs { promote: false } }
I20240502 20:45:46.664404 3740 raft_consensus.cc:1261] T 63e2878c2981429cbcb41837f3f0c44a P d740107bba7b4fa290be7e085ddd55c6 [term 1 LEARNER]: Refusing update from remote peer 33be9a5b29214a8d84c2fd7a4936a920: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:45:46.664810 4009 consensus_queue.cc:1035] T 63e2878c2981429cbcb41837f3f0c44a P 33be9a5b29214a8d84c2fd7a4936a920 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d740107bba7b4fa290be7e085ddd55c6" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 38655 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:45:46.666550 4010 raft_consensus.cc:2932] T 63e2878c2981429cbcb41837f3f0c44a P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d740107bba7b4fa290be7e085ddd55c6 (127.30.32.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "d740107bba7b4fa290be7e085ddd55c6" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 38655 } attrs { promote: false } } }
I20240502 20:45:46.666975 3740 raft_consensus.cc:2932] T 63e2878c2981429cbcb41837f3f0c44a P d740107bba7b4fa290be7e085ddd55c6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d740107bba7b4fa290be7e085ddd55c6 (127.30.32.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "d740107bba7b4fa290be7e085ddd55c6" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 38655 } attrs { promote: false } } }
I20240502 20:45:46.668999 746 catalog_manager.cc:5484] T 63e2878c2981429cbcb41837f3f0c44a P 33be9a5b29214a8d84c2fd7a4936a920 reported cstate change: config changed from index 4 to 5, d740107bba7b4fa290be7e085ddd55c6 (127.30.32.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "33be9a5b29214a8d84c2fd7a4936a920" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d740107bba7b4fa290be7e085ddd55c6" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 38655 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:46.693264 3948 raft_consensus.cc:979] T 8d06cc46e1b04a809b88c070996ba1ed P ed5783865eb74d39bee2a38e9139dea4: : Instructing follower d740107bba7b4fa290be7e085ddd55c6 to start an election
I20240502 20:45:46.693399 3948 raft_consensus.cc:1067] T 8d06cc46e1b04a809b88c070996ba1ed P ed5783865eb74d39bee2a38e9139dea4 [term 1 LEADER]: Signalling peer d740107bba7b4fa290be7e085ddd55c6 to start an election
I20240502 20:45:46.693890 3740 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8d06cc46e1b04a809b88c070996ba1ed"
dest_uuid: "d740107bba7b4fa290be7e085ddd55c6"
from {username='slave'} at 127.30.32.66:59021
I20240502 20:45:46.694092 3740 raft_consensus.cc:482] T 8d06cc46e1b04a809b88c070996ba1ed P d740107bba7b4fa290be7e085ddd55c6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:45:46.694202 3740 raft_consensus.cc:3037] T 8d06cc46e1b04a809b88c070996ba1ed P d740107bba7b4fa290be7e085ddd55c6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:46.695035 3740 raft_consensus.cc:504] T 8d06cc46e1b04a809b88c070996ba1ed P d740107bba7b4fa290be7e085ddd55c6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } } peers { permanent_uuid: "d740107bba7b4fa290be7e085ddd55c6" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 38655 } attrs { promote: false } }
I20240502 20:45:46.695909 3740 leader_election.cc:288] T 8d06cc46e1b04a809b88c070996ba1ed P d740107bba7b4fa290be7e085ddd55c6 [CANDIDATE]: Term 2 election: Requested vote from peers ed5783865eb74d39bee2a38e9139dea4 (127.30.32.66:45113)
I20240502 20:45:46.696323 1024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d06cc46e1b04a809b88c070996ba1ed" candidate_uuid: "d740107bba7b4fa290be7e085ddd55c6" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "ed5783865eb74d39bee2a38e9139dea4"
I20240502 20:45:46.696431 1024 raft_consensus.cc:3032] T 8d06cc46e1b04a809b88c070996ba1ed P ed5783865eb74d39bee2a38e9139dea4 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:45:46.696480 1024 raft_consensus.cc:729] T 8d06cc46e1b04a809b88c070996ba1ed P ed5783865eb74d39bee2a38e9139dea4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ed5783865eb74d39bee2a38e9139dea4, State: Running, Role: LEADER
I20240502 20:45:46.696635 1024 consensus_queue.cc:260] T 8d06cc46e1b04a809b88c070996ba1ed P ed5783865eb74d39bee2a38e9139dea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } } peers { permanent_uuid: "d740107bba7b4fa290be7e085ddd55c6" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 38655 } attrs { promote: false } }
I20240502 20:45:46.696767 1024 raft_consensus.cc:3037] T 8d06cc46e1b04a809b88c070996ba1ed P ed5783865eb74d39bee2a38e9139dea4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:46.697594 1024 raft_consensus.cc:2454] T 8d06cc46e1b04a809b88c070996ba1ed P ed5783865eb74d39bee2a38e9139dea4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d740107bba7b4fa290be7e085ddd55c6 in term 2.
I20240502 20:45:46.697994 3676 leader_election.cc:302] T 8d06cc46e1b04a809b88c070996ba1ed P d740107bba7b4fa290be7e085ddd55c6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: d740107bba7b4fa290be7e085ddd55c6, ed5783865eb74d39bee2a38e9139dea4; no voters:
I20240502 20:45:46.698207 3922 raft_consensus.cc:2781] T 8d06cc46e1b04a809b88c070996ba1ed P d740107bba7b4fa290be7e085ddd55c6 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:45:46.698398 3922 raft_consensus.cc:686] T 8d06cc46e1b04a809b88c070996ba1ed P d740107bba7b4fa290be7e085ddd55c6 [term 2 LEADER]: Becoming Leader. State: Replica: d740107bba7b4fa290be7e085ddd55c6, State: Running, Role: LEADER
I20240502 20:45:46.698684 3922 consensus_queue.cc:237] T 8d06cc46e1b04a809b88c070996ba1ed P d740107bba7b4fa290be7e085ddd55c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } } peers { permanent_uuid: "d740107bba7b4fa290be7e085ddd55c6" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 38655 } attrs { promote: false } }
I20240502 20:45:46.701210 743 catalog_manager.cc:5484] T 8d06cc46e1b04a809b88c070996ba1ed P d740107bba7b4fa290be7e085ddd55c6 reported cstate change: term changed from 1 to 2, leader changed from ed5783865eb74d39bee2a38e9139dea4 (127.30.32.66) to d740107bba7b4fa290be7e085ddd55c6 (127.30.32.72). New cstate: current_term: 2 leader_uuid: "d740107bba7b4fa290be7e085ddd55c6" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d740107bba7b4fa290be7e085ddd55c6" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 38655 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 20:45:46.707003 738 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 8d06cc46e1b04a809b88c070996ba1ed with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader d740107bba7b4fa290be7e085ddd55c6 (127.30.32.72:38655) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20240502 20:45:46.797070 893 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "63e2878c2981429cbcb41837f3f0c44a"
dest_uuid: "33be9a5b29214a8d84c2fd7a4936a920"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38116
I20240502 20:45:46.797232 893 raft_consensus.cc:595] T 63e2878c2981429cbcb41837f3f0c44a P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 LEADER]: Received request to transfer leadership
I20240502 20:45:47.014458 893 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "63e2878c2981429cbcb41837f3f0c44a"
dest_uuid: "33be9a5b29214a8d84c2fd7a4936a920"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38130
I20240502 20:45:47.014603 893 raft_consensus.cc:595] T 63e2878c2981429cbcb41837f3f0c44a P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 LEADER]: Received request to transfer leadership
I20240502 20:45:47.067441 4013 raft_consensus.cc:979] T 63e2878c2981429cbcb41837f3f0c44a P 33be9a5b29214a8d84c2fd7a4936a920: : Instructing follower d740107bba7b4fa290be7e085ddd55c6 to start an election
I20240502 20:45:47.067581 4013 raft_consensus.cc:1067] T 63e2878c2981429cbcb41837f3f0c44a P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 LEADER]: Signalling peer d740107bba7b4fa290be7e085ddd55c6 to start an election
I20240502 20:45:47.068087 3740 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "63e2878c2981429cbcb41837f3f0c44a"
dest_uuid: "d740107bba7b4fa290be7e085ddd55c6"
from {username='slave'} at 127.30.32.65:52313
I20240502 20:45:47.068260 3740 raft_consensus.cc:482] T 63e2878c2981429cbcb41837f3f0c44a P d740107bba7b4fa290be7e085ddd55c6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:45:47.068337 3740 raft_consensus.cc:3037] T 63e2878c2981429cbcb41837f3f0c44a P d740107bba7b4fa290be7e085ddd55c6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:47.069483 3740 raft_consensus.cc:504] T 63e2878c2981429cbcb41837f3f0c44a P d740107bba7b4fa290be7e085ddd55c6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "d740107bba7b4fa290be7e085ddd55c6" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 38655 } attrs { promote: false } }
I20240502 20:45:47.069840 3740 leader_election.cc:288] T 63e2878c2981429cbcb41837f3f0c44a P d740107bba7b4fa290be7e085ddd55c6 [CANDIDATE]: Term 2 election: Requested vote from peers 33be9a5b29214a8d84c2fd7a4936a920 (127.30.32.65:45537)
I20240502 20:45:47.070348 893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63e2878c2981429cbcb41837f3f0c44a" candidate_uuid: "d740107bba7b4fa290be7e085ddd55c6" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "33be9a5b29214a8d84c2fd7a4936a920"
I20240502 20:45:47.070467 893 raft_consensus.cc:3032] T 63e2878c2981429cbcb41837f3f0c44a P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:45:47.070521 893 raft_consensus.cc:729] T 63e2878c2981429cbcb41837f3f0c44a P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 33be9a5b29214a8d84c2fd7a4936a920, State: Running, Role: LEADER
I20240502 20:45:47.070672 893 consensus_queue.cc:260] T 63e2878c2981429cbcb41837f3f0c44a P 33be9a5b29214a8d84c2fd7a4936a920 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "d740107bba7b4fa290be7e085ddd55c6" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 38655 } attrs { promote: false } }
I20240502 20:45:47.070801 893 raft_consensus.cc:3037] T 63e2878c2981429cbcb41837f3f0c44a P 33be9a5b29214a8d84c2fd7a4936a920 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:47.071591 893 raft_consensus.cc:2454] T 63e2878c2981429cbcb41837f3f0c44a P 33be9a5b29214a8d84c2fd7a4936a920 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d740107bba7b4fa290be7e085ddd55c6 in term 2.
I20240502 20:45:47.071925 3673 leader_election.cc:302] T 63e2878c2981429cbcb41837f3f0c44a P d740107bba7b4fa290be7e085ddd55c6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 33be9a5b29214a8d84c2fd7a4936a920, d740107bba7b4fa290be7e085ddd55c6; no voters:
I20240502 20:45:47.072131 3922 raft_consensus.cc:2781] T 63e2878c2981429cbcb41837f3f0c44a P d740107bba7b4fa290be7e085ddd55c6 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:45:47.072247 3922 raft_consensus.cc:686] T 63e2878c2981429cbcb41837f3f0c44a P d740107bba7b4fa290be7e085ddd55c6 [term 2 LEADER]: Becoming Leader. State: Replica: d740107bba7b4fa290be7e085ddd55c6, State: Running, Role: LEADER
I20240502 20:45:47.072402 3922 consensus_queue.cc:237] T 63e2878c2981429cbcb41837f3f0c44a P d740107bba7b4fa290be7e085ddd55c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } } peers { permanent_uuid: "d740107bba7b4fa290be7e085ddd55c6" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 38655 } attrs { promote: false } }
I20240502 20:45:47.073904 743 catalog_manager.cc:5484] T 63e2878c2981429cbcb41837f3f0c44a P d740107bba7b4fa290be7e085ddd55c6 reported cstate change: term changed from 1 to 2, leader changed from 33be9a5b29214a8d84c2fd7a4936a920 (127.30.32.65) to d740107bba7b4fa290be7e085ddd55c6 (127.30.32.72). New cstate: current_term: 2 leader_uuid: "d740107bba7b4fa290be7e085ddd55c6" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d740107bba7b4fa290be7e085ddd55c6" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 38655 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:47.236315 1024 raft_consensus.cc:1261] T 8d06cc46e1b04a809b88c070996ba1ed P ed5783865eb74d39bee2a38e9139dea4 [term 2 FOLLOWER]: Refusing update from remote peer d740107bba7b4fa290be7e085ddd55c6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 20:45:47.236824 3922 consensus_queue.cc:1035] T 8d06cc46e1b04a809b88c070996ba1ed P d740107bba7b4fa290be7e085ddd55c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed5783865eb74d39bee2a38e9139dea4" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 45113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:45:47.332100 3740 consensus_queue.cc:237] T 8d06cc46e1b04a809b88c070996ba1ed P d740107bba7b4fa290be7e085ddd55c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d740107bba7b4fa290be7e085ddd55c6" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 38655 } attrs { promote: false } }
I20240502 20:45:47.332818 4014 raft_consensus.cc:2932] T 8d06cc46e1b04a809b88c070996ba1ed P d740107bba7b4fa290be7e085ddd55c6 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER ed5783865eb74d39bee2a38e9139dea4 (127.30.32.66) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d740107bba7b4fa290be7e085ddd55c6" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 38655 } attrs { promote: false } } }
I20240502 20:45:47.334324 738 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8d06cc46e1b04a809b88c070996ba1ed with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 20:45:47.335219 746 catalog_manager.cc:5484] T 8d06cc46e1b04a809b88c070996ba1ed P d740107bba7b4fa290be7e085ddd55c6 reported cstate change: config changed from index 5 to 7, VOTER ed5783865eb74d39bee2a38e9139dea4 (127.30.32.66) evicted. New cstate: current_term: 2 leader_uuid: "d740107bba7b4fa290be7e085ddd55c6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d740107bba7b4fa290be7e085ddd55c6" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 38655 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:47.337774 1003 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d06cc46e1b04a809b88c070996ba1ed with delete_type TABLET_DATA_TOMBSTONED (TS ed5783865eb74d39bee2a38e9139dea4 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57638
I20240502 20:45:47.338122 4060 tablet_replica.cc:330] T 8d06cc46e1b04a809b88c070996ba1ed P ed5783865eb74d39bee2a38e9139dea4: stopping tablet replica
I20240502 20:45:47.338265 4060 raft_consensus.cc:2229] T 8d06cc46e1b04a809b88c070996ba1ed P ed5783865eb74d39bee2a38e9139dea4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:47.338382 4060 raft_consensus.cc:2258] T 8d06cc46e1b04a809b88c070996ba1ed P ed5783865eb74d39bee2a38e9139dea4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:47.339767 4060 ts_tablet_manager.cc:1875] T 8d06cc46e1b04a809b88c070996ba1ed P ed5783865eb74d39bee2a38e9139dea4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:45:47.341745 4060 ts_tablet_manager.cc:1888] T 8d06cc46e1b04a809b88c070996ba1ed P ed5783865eb74d39bee2a38e9139dea4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 20:45:47.341825 4060 log.cc:1196] T 8d06cc46e1b04a809b88c070996ba1ed P ed5783865eb74d39bee2a38e9139dea4: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal/wals/8d06cc46e1b04a809b88c070996ba1ed
I20240502 20:45:47.342393 738 catalog_manager.cc:4830] TS ed5783865eb74d39bee2a38e9139dea4 (127.30.32.66:45113): tablet 8d06cc46e1b04a809b88c070996ba1ed (table rebalance_test_table_2 [id=fddf24b281f54ee394d7a819baadce47]) successfully deleted
I20240502 20:45:47.657328 893 raft_consensus.cc:1261] T 63e2878c2981429cbcb41837f3f0c44a P 33be9a5b29214a8d84c2fd7a4936a920 [term 2 FOLLOWER]: Refusing update from remote peer d740107bba7b4fa290be7e085ddd55c6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 20:45:47.657776 4014 consensus_queue.cc:1035] T 63e2878c2981429cbcb41837f3f0c44a P d740107bba7b4fa290be7e085ddd55c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33be9a5b29214a8d84c2fd7a4936a920" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 45537 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:45:47.706468 3740 consensus_queue.cc:237] T 63e2878c2981429cbcb41837f3f0c44a P d740107bba7b4fa290be7e085ddd55c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d740107bba7b4fa290be7e085ddd55c6" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 38655 } attrs { promote: false } }
I20240502 20:45:47.707186 3922 raft_consensus.cc:2932] T 63e2878c2981429cbcb41837f3f0c44a P d740107bba7b4fa290be7e085ddd55c6 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 33be9a5b29214a8d84c2fd7a4936a920 (127.30.32.65) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d740107bba7b4fa290be7e085ddd55c6" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 38655 } attrs { promote: false } } }
I20240502 20:45:47.708557 738 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 63e2878c2981429cbcb41837f3f0c44a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 20:45:47.709520 746 catalog_manager.cc:5484] T 63e2878c2981429cbcb41837f3f0c44a P d740107bba7b4fa290be7e085ddd55c6 reported cstate change: config changed from index 5 to 7, VOTER 33be9a5b29214a8d84c2fd7a4936a920 (127.30.32.65) evicted. New cstate: current_term: 2 leader_uuid: "d740107bba7b4fa290be7e085ddd55c6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d740107bba7b4fa290be7e085ddd55c6" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 38655 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:47.711827 873 tablet_service.cc:1514] Processing DeleteTablet for tablet 63e2878c2981429cbcb41837f3f0c44a with delete_type TABLET_DATA_TOMBSTONED (TS 33be9a5b29214a8d84c2fd7a4936a920 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:54242
I20240502 20:45:47.712195 4083 tablet_replica.cc:330] T 63e2878c2981429cbcb41837f3f0c44a P 33be9a5b29214a8d84c2fd7a4936a920: stopping tablet replica
I20240502 20:45:47.712332 4083 raft_consensus.cc:2229] T 63e2878c2981429cbcb41837f3f0c44a P 33be9a5b29214a8d84c2fd7a4936a920 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:47.712446 4083 raft_consensus.cc:2258] T 63e2878c2981429cbcb41837f3f0c44a P 33be9a5b29214a8d84c2fd7a4936a920 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:47.713979 4083 ts_tablet_manager.cc:1875] T 63e2878c2981429cbcb41837f3f0c44a P 33be9a5b29214a8d84c2fd7a4936a920: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:45:47.715925 4083 ts_tablet_manager.cc:1888] T 63e2878c2981429cbcb41837f3f0c44a P 33be9a5b29214a8d84c2fd7a4936a920: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 20:45:47.715999 4083 log.cc:1196] T 63e2878c2981429cbcb41837f3f0c44a P 33be9a5b29214a8d84c2fd7a4936a920: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal/wals/63e2878c2981429cbcb41837f3f0c44a
I20240502 20:45:47.716529 735 catalog_manager.cc:4830] TS 33be9a5b29214a8d84c2fd7a4936a920 (127.30.32.65:45537): tablet 63e2878c2981429cbcb41837f3f0c44a (table rebalance_test_table_4 [id=1c8687d7588245508f9c47ea918ebc6a]) successfully deleted
I20240502 20:45:47.920519 2408 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:47.920763 873 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:47.927183 1003 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:47.929767 2539 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:47.930819 2277 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:47.937165 2137 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:47.937175 3720 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:47.961962 2005 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:48.039045 2277 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:48.042728 2137 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:48.042886 2539 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:48.043309 2408 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:48.052950 1003 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:48.058571 3720 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:48.061918 873 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:48.072221 2005 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:48.159365 2277 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:48.160456 2539 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:48.165496 1003 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:48.165925 3720 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:48.171833 873 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:48.185882 2408 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:48.195695 2137 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:45:48.196866 2005 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
5c223b22641f4581b6ec8c5e553014cd | 127.30.32.126:43021 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.30.32.84:35991 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
0dfc68b63d15462398c76bfbff0e3cf8 | 127.30.32.71:40439 | HEALTHY | <none> | 2 | 0
2260afde4ccb498189d7b25c00a2c727 | 127.30.32.68:40723 | HEALTHY | <none> | 2 | 0
33be9a5b29214a8d84c2fd7a4936a920 | 127.30.32.65:45537 | HEALTHY | <none> | 2 | 0
6d231fa7f1684535af46a9b196111880 | 127.30.32.70:43291 | HEALTHY | <none> | 2 | 0
cec988e98db64e2f98b030efc16d4108 | 127.30.32.69:34789 | HEALTHY | <none> | 2 | 0
d740107bba7b4fa290be7e085ddd55c6 | 127.30.32.72:38655 | HEALTHY | <none> | 2 | 0
ed5783865eb74d39bee2a38e9139dea4 | 127.30.32.66:45113 | HEALTHY | <none> | 2 | 0
f7fd7bf659f84ae184696b891db5221a | 127.30.32.67:42407 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.30.32.65 | experimental | 127.30.32.65:45537
local_ip_for_outbound_sockets | 127.30.32.66 | experimental | 127.30.32.66:45113
local_ip_for_outbound_sockets | 127.30.32.67 | experimental | 127.30.32.67:42407
local_ip_for_outbound_sockets | 127.30.32.68 | experimental | 127.30.32.68:40723
local_ip_for_outbound_sockets | 127.30.32.69 | experimental | 127.30.32.69:34789
local_ip_for_outbound_sockets | 127.30.32.70 | experimental | 127.30.32.70:43291
local_ip_for_outbound_sockets | 127.30.32.71 | experimental | 127.30.32.71:40439
local_ip_for_outbound_sockets | 127.30.32.72 | experimental | 127.30.32.72:38655
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.30.32.65:45537
server_dump_info_path | /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.30.32.66:45113
server_dump_info_path | /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.30.32.67:42407
server_dump_info_path | /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.30.32.68:40723
server_dump_info_path | /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.30.32.69:34789
server_dump_info_path | /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.30.32.70:43291
server_dump_info_path | /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.30.32.71:40439
server_dump_info_path | /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.30.32.72:38655
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.30.32.84:35991 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 2
Maximum | 2
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 20:45:48.218537 30849 log_verifier.cc:126] Checking tablet 0734f8b2eeb04ad99ff827c0ecf90663
I20240502 20:45:48.225275 30849 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 0734f8b2eeb04ad99ff827c0ecf90663
I20240502 20:45:48.225325 30849 log_verifier.cc:126] Checking tablet 21a27f13d4104a8a87aa8b935c6c8dfa
I20240502 20:45:48.231945 30849 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 21a27f13d4104a8a87aa8b935c6c8dfa
I20240502 20:45:48.231997 30849 log_verifier.cc:126] Checking tablet 49eb89bf68d942398c68e2bf1d0f6370
I20240502 20:45:48.237924 30849 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 49eb89bf68d942398c68e2bf1d0f6370
I20240502 20:45:48.237978 30849 log_verifier.cc:126] Checking tablet 63e2878c2981429cbcb41837f3f0c44a
I20240502 20:45:48.246328 30849 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 63e2878c2981429cbcb41837f3f0c44a
I20240502 20:45:48.246395 30849 log_verifier.cc:126] Checking tablet 74b7ff29e276451985e7cbe72ceff148
I20240502 20:45:48.252189 30849 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 74b7ff29e276451985e7cbe72ceff148
I20240502 20:45:48.252261 30849 log_verifier.cc:126] Checking tablet 7a888b6096c2427f8a7f67b26c3c00b7
I20240502 20:45:48.257980 30849 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 7a888b6096c2427f8a7f67b26c3c00b7
I20240502 20:45:48.258035 30849 log_verifier.cc:126] Checking tablet 83593d702ac542c28bd1721338a55c5c
I20240502 20:45:48.263433 30849 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 83593d702ac542c28bd1721338a55c5c
I20240502 20:45:48.263484 30849 log_verifier.cc:126] Checking tablet 89fc553dd07e4af5bd37801a34c4ac66
I20240502 20:45:48.268831 30849 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 89fc553dd07e4af5bd37801a34c4ac66
I20240502 20:45:48.268882 30849 log_verifier.cc:126] Checking tablet 8d06cc46e1b04a809b88c070996ba1ed
I20240502 20:45:48.276952 30849 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 8d06cc46e1b04a809b88c070996ba1ed
I20240502 20:45:48.277031 30849 log_verifier.cc:126] Checking tablet 975b1d549985459da90b6cde03b0c5b8
I20240502 20:45:48.282905 30849 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 975b1d549985459da90b6cde03b0c5b8
I20240502 20:45:48.282964 30849 log_verifier.cc:126] Checking tablet a6c9b839b6a346aca42a280aed623913
I20240502 20:45:48.289166 30849 log_verifier.cc:177] Verified matching terms for 7 ops in tablet a6c9b839b6a346aca42a280aed623913
I20240502 20:45:48.289225 30849 log_verifier.cc:126] Checking tablet afc220634a164930bdaa8ece818fb2e7
I20240502 20:45:48.295176 30849 log_verifier.cc:177] Verified matching terms for 7 ops in tablet afc220634a164930bdaa8ece818fb2e7
I20240502 20:45:48.295234 30849 log_verifier.cc:126] Checking tablet cf7de28bcb20418f99846c121cd88531
I20240502 20:45:48.300812 30849 log_verifier.cc:177] Verified matching terms for 7 ops in tablet cf7de28bcb20418f99846c121cd88531
I20240502 20:45:48.300868 30849 log_verifier.cc:126] Checking tablet da05217d82a74bd6aeb54330c3d6d370
I20240502 20:45:48.306516 30849 log_verifier.cc:177] Verified matching terms for 7 ops in tablet da05217d82a74bd6aeb54330c3d6d370
I20240502 20:45:48.306564 30849 log_verifier.cc:126] Checking tablet e7080f04f87d4a6f926edea263b450bc
I20240502 20:45:48.311131 30849 log_verifier.cc:177] Verified matching terms for 2 ops in tablet e7080f04f87d4a6f926edea263b450bc
I20240502 20:45:48.311175 30849 log_verifier.cc:126] Checking tablet ff0d71af3fd4483caa41325903f9ac39
I20240502 20:45:48.316582 30849 log_verifier.cc:177] Verified matching terms for 3 ops in tablet ff0d71af3fd4483caa41325903f9ac39
I20240502 20:45:48.317044 30849 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskolF7xA/build/debug/bin/kudu with pid 811
I20240502 20:45:48.324023 30849 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskolF7xA/build/debug/bin/kudu with pid 942
I20240502 20:45:48.330272 30849 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskolF7xA/build/debug/bin/kudu with pid 1943
I20240502 20:45:48.335903 30849 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskolF7xA/build/debug/bin/kudu with pid 2074
I20240502 20:45:48.341317 30849 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskolF7xA/build/debug/bin/kudu with pid 2207
I20240502 20:45:48.346915 30849 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskolF7xA/build/debug/bin/kudu with pid 2346
I20240502 20:45:48.352324 30849 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskolF7xA/build/debug/bin/kudu with pid 2477
I20240502 20:45:48.357669 30849 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskolF7xA/build/debug/bin/kudu with pid 3658
I20240502 20:45:48.362870 30849 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskolF7xA/build/debug/bin/kudu with pid 720
2024-05-02T20:45:48Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/1 (15882 ms)
[----------] 1 test from RebalanceParamTest (15882 ms total)
[----------] 1 test from DDLDuringRebalancingTest
[ RUN ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/1
I20240502 20:45:48.401932 30849 test_util.cc:268] Using random seed: -1340147451
I20240502 20:45:48.402738 30849 ts_itest-base.cc:112] Starting cluster with:
I20240502 20:45:48.402786 30849 ts_itest-base.cc:113] --------------
I20240502 20:45:48.402806 30849 ts_itest-base.cc:114] 8 tablet servers
I20240502 20:45:48.402824 30849 ts_itest-base.cc:115] 3 replicas per TS
I20240502 20:45:48.402853 30849 ts_itest-base.cc:116] --------------
2024-05-02T20:45:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T20:45:48Z Disabled control of system clock
I20240502 20:45:48.411444 30849 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskolF7xA/build/debug/bin/kudu
/tmp/dist-test-taskolF7xA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.32.126:40167
--webserver_interface=127.30.32.126
--webserver_port=0
--builtin_ntp_servers=127.30.32.84:39781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.32.126:40167
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240502 20:45:48.516278 4204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:48.516479 4204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:48.516517 4204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:48.519490 4204 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:48.519551 4204 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 20:45:48.519579 4204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:48.519618 4204 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 20:45:48.519649 4204 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 20:45:48.523236 4204 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.32.84:39781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.30.32.126:40167
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.32.126:40167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.30.32.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.4204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:48.524050 4204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:48.525175 4204 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:48.532797 4212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:48.533491 4210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:48.534101 4209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:48.534816 4204 server_base.cc:1034] running on GCE node
I20240502 20:45:48.535270 4204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:48.536128 4204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:48.537273 4204 hybrid_clock.cc:648] HybridClock initialized: now 1714682748537260 us; error 27 us; skew 500 ppm
I20240502 20:45:48.539098 4204 webserver.cc:460] Webserver started at http://127.30.32.126:36585/ using document root <none> and password file <none>
I20240502 20:45:48.539602 4204 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:48.539654 4204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:48.539819 4204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:48.541432 4204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "4c527b25ee464801a057bbad1ac95aec"
format_stamp: "Formatted at 2024-05-02 20:45:48 on dist-test-slave-6336"
I20240502 20:45:48.541872 4204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "4c527b25ee464801a057bbad1ac95aec"
format_stamp: "Formatted at 2024-05-02 20:45:48 on dist-test-slave-6336"
I20240502 20:45:48.545022 4204 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 20:45:48.547049 4218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:48.547983 4204 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:48.548099 4204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/wal
uuid: "4c527b25ee464801a057bbad1ac95aec"
format_stamp: "Formatted at 2024-05-02 20:45:48 on dist-test-slave-6336"
I20240502 20:45:48.548178 4204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:48.559899 4204 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:48.560418 4204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:48.560560 4204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:48.566947 4204 rpc_server.cc:307] RPC server started. Bound to: 127.30.32.126:40167
I20240502 20:45:48.566959 4270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.32.126:40167 every 8 connection(s)
I20240502 20:45:48.567824 4204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 20:45:48.570247 4271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:48.574961 4271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: Bootstrap starting.
I20240502 20:45:48.576504 30849 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskolF7xA/build/debug/bin/kudu as pid 4204
I20240502 20:45:48.576617 30849 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 20:45:48.577116 4271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:48.577942 4271 log.cc:824] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:48.579453 4271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: No bootstrap required, opened a new log
I20240502 20:45:48.582357 4271 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c527b25ee464801a057bbad1ac95aec" member_type: VOTER last_known_addr { host: "127.30.32.126" port: 40167 } }
I20240502 20:45:48.582561 4271 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:48.582605 4271 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c527b25ee464801a057bbad1ac95aec, State: Initialized, Role: FOLLOWER
I20240502 20:45:48.583021 4271 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c527b25ee464801a057bbad1ac95aec" member_type: VOTER last_known_addr { host: "127.30.32.126" port: 40167 } }
I20240502 20:45:48.583173 4271 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 20:45:48.583248 4271 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 20:45:48.583372 4271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:48.584029 4271 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c527b25ee464801a057bbad1ac95aec" member_type: VOTER last_known_addr { host: "127.30.32.126" port: 40167 } }
I20240502 20:45:48.584362 4271 leader_election.cc:302] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c527b25ee464801a057bbad1ac95aec; no voters:
I20240502 20:45:48.584635 4271 leader_election.cc:288] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 20:45:48.585040 4276 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:48.585222 4276 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec [term 1 LEADER]: Becoming Leader. State: Replica: 4c527b25ee464801a057bbad1ac95aec, State: Running, Role: LEADER
I20240502 20:45:48.585510 4276 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c527b25ee464801a057bbad1ac95aec" member_type: VOTER last_known_addr { host: "127.30.32.126" port: 40167 } }
I20240502 20:45:48.585702 4271 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec [sys.catalog]: configured and running, proceeding with master startup.
I20240502 20:45:48.586452 4276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c527b25ee464801a057bbad1ac95aec. Latest consensus state: current_term: 1 leader_uuid: "4c527b25ee464801a057bbad1ac95aec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c527b25ee464801a057bbad1ac95aec" member_type: VOTER last_known_addr { host: "127.30.32.126" port: 40167 } } }
I20240502 20:45:48.586582 4276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec [sys.catalog]: This master's current role is: LEADER
I20240502 20:45:48.586782 4277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c527b25ee464801a057bbad1ac95aec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c527b25ee464801a057bbad1ac95aec" member_type: VOTER last_known_addr { host: "127.30.32.126" port: 40167 } } }
I20240502 20:45:48.586874 4277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec [sys.catalog]: This master's current role is: LEADER
I20240502 20:45:48.586970 4285 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 20:45:48.589687 4285 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 20:45:48.594182 4285 catalog_manager.cc:1347] Generated new cluster ID: f06c3382a6f54525827ae97e4c5e2810
I20240502 20:45:48.594249 4285 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 20:45:48.653725 4285 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 20:45:48.654932 4285 catalog_manager.cc:1504] Loading token signing keys...
I20240502 20:45:48.663619 4285 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: Generated new TSK 0
I20240502 20:45:48.664258 4285 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 20:45:48.670917 30849 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskolF7xA/build/debug/bin/kudu
/tmp/dist-test-taskolF7xA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.32.65:0
--local_ip_for_outbound_sockets=127.30.32.65
--webserver_interface=127.30.32.65
--webserver_port=0
--tserver_master_addrs=127.30.32.126:40167
--builtin_ntp_servers=127.30.32.84:39781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 20:45:48.780354 4295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:48.780570 4295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:48.780613 4295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:48.783576 4295 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:48.783638 4295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:48.783736 4295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.32.65
I20240502 20:45:48.787382 4295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.32.84:39781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.32.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.30.32.65
--webserver_port=0
--tserver_master_addrs=127.30.32.126:40167
--never_fsync=true
--heap_profile_path=/tmp/kudu.4295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.32.65
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:48.788282 4295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:48.789412 4295 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:48.798002 4300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:48.798677 4301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:48.799401 4303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:48.800105 4295 server_base.cc:1034] running on GCE node
I20240502 20:45:48.800530 4295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:48.801133 4295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:48.802291 4295 hybrid_clock.cc:648] HybridClock initialized: now 1714682748802272 us; error 36 us; skew 500 ppm
I20240502 20:45:48.804498 4295 webserver.cc:460] Webserver started at http://127.30.32.65:46179/ using document root <none> and password file <none>
I20240502 20:45:48.805142 4295 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:48.805212 4295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:48.805433 4295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:48.807163 4295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "c92e6e46782849479c8a0c6d82276fb2"
format_stamp: "Formatted at 2024-05-02 20:45:48 on dist-test-slave-6336"
I20240502 20:45:48.807679 4295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "c92e6e46782849479c8a0c6d82276fb2"
format_stamp: "Formatted at 2024-05-02 20:45:48 on dist-test-slave-6336"
I20240502 20:45:48.811205 4295 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20240502 20:45:48.813606 4309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:48.814661 4295 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 20:45:48.814772 4295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "c92e6e46782849479c8a0c6d82276fb2"
format_stamp: "Formatted at 2024-05-02 20:45:48 on dist-test-slave-6336"
I20240502 20:45:48.814893 4295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:48.828466 4295 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:48.829108 4295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:48.829272 4295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:48.829823 4295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:48.830722 4295 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:48.830772 4295 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:48.830819 4295 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:48.830848 4295 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:48.839138 4295 rpc_server.cc:307] RPC server started. Bound to: 127.30.32.65:33827
I20240502 20:45:48.839182 4422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.32.65:33827 every 8 connection(s)
I20240502 20:45:48.840128 4295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 20:45:48.847249 30849 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskolF7xA/build/debug/bin/kudu as pid 4295
I20240502 20:45:48.847406 30849 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 20:45:48.848973 4423 heartbeater.cc:347] Connected to a master server at 127.30.32.126:40167
I20240502 20:45:48.849242 4423 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:48.849792 4423 heartbeater.cc:511] Master 127.30.32.126:40167 requested a full tablet report, sending...
I20240502 20:45:48.849859 30849 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskolF7xA/build/debug/bin/kudu
/tmp/dist-test-taskolF7xA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.32.66:0
--local_ip_for_outbound_sockets=127.30.32.66
--webserver_interface=127.30.32.66
--webserver_port=0
--tserver_master_addrs=127.30.32.126:40167
--builtin_ntp_servers=127.30.32.84:39781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 20:45:48.851245 4234 ts_manager.cc:194] Registered new tserver with Master: c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:45:48.852473 4234 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.32.65:57733
W20240502 20:45:48.956081 4426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:48.956296 4426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:48.956343 4426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:48.959412 4426 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:48.959486 4426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:48.959587 4426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.32.66
I20240502 20:45:48.963160 4426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.32.84:39781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.32.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.30.32.66
--webserver_port=0
--tserver_master_addrs=127.30.32.126:40167
--never_fsync=true
--heap_profile_path=/tmp/kudu.4426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.32.66
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:48.964030 4426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:48.965189 4426 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:48.973668 4434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:48.974400 4431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:48.975106 4432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:48.975811 4426 server_base.cc:1034] running on GCE node
I20240502 20:45:48.976204 4426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:48.976759 4426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:48.977906 4426 hybrid_clock.cc:648] HybridClock initialized: now 1714682748977879 us; error 39 us; skew 500 ppm
I20240502 20:45:48.979780 4426 webserver.cc:460] Webserver started at http://127.30.32.66:44845/ using document root <none> and password file <none>
I20240502 20:45:48.980321 4426 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:48.980378 4426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:48.980557 4426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:48.982208 4426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "c3d26eafd51547acb780d4aac5ebe7cc"
format_stamp: "Formatted at 2024-05-02 20:45:48 on dist-test-slave-6336"
I20240502 20:45:48.982656 4426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "c3d26eafd51547acb780d4aac5ebe7cc"
format_stamp: "Formatted at 2024-05-02 20:45:48 on dist-test-slave-6336"
I20240502 20:45:48.985905 4426 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 20:45:48.988020 4440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:48.988952 4426 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:48.989086 4426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "c3d26eafd51547acb780d4aac5ebe7cc"
format_stamp: "Formatted at 2024-05-02 20:45:48 on dist-test-slave-6336"
I20240502 20:45:48.989166 4426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:49.004165 4426 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:49.004719 4426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:49.004869 4426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:49.005414 4426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:49.006268 4426 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:49.006314 4426 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:49.006361 4426 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:49.006390 4426 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:49.014350 4426 rpc_server.cc:307] RPC server started. Bound to: 127.30.32.66:42097
I20240502 20:45:49.014396 4553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.32.66:42097 every 8 connection(s)
I20240502 20:45:49.015270 4426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 20:45:49.023623 4554 heartbeater.cc:347] Connected to a master server at 127.30.32.126:40167
I20240502 20:45:49.023860 4554 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:49.024374 4554 heartbeater.cc:511] Master 127.30.32.126:40167 requested a full tablet report, sending...
I20240502 20:45:49.025415 4234 ts_manager.cc:194] Registered new tserver with Master: c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097)
I20240502 20:45:49.025527 30849 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskolF7xA/build/debug/bin/kudu as pid 4426
I20240502 20:45:49.025650 30849 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 20:45:49.026122 4234 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.32.66:38757
I20240502 20:45:49.027714 30849 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskolF7xA/build/debug/bin/kudu
/tmp/dist-test-taskolF7xA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.32.67:0
--local_ip_for_outbound_sockets=127.30.32.67
--webserver_interface=127.30.32.67
--webserver_port=0
--tserver_master_addrs=127.30.32.126:40167
--builtin_ntp_servers=127.30.32.84:39781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 20:45:49.138024 4557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:49.138250 4557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:49.138303 4557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:49.141319 4557 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:49.141378 4557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:49.141484 4557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.32.67
I20240502 20:45:49.145080 4557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.32.84:39781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.32.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.30.32.67
--webserver_port=0
--tserver_master_addrs=127.30.32.126:40167
--never_fsync=true
--heap_profile_path=/tmp/kudu.4557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.32.67
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:49.145961 4557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:49.147053 4557 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:49.155540 4565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:49.155982 4562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:49.156667 4563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:49.157406 4557 server_base.cc:1034] running on GCE node
I20240502 20:45:49.157786 4557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:49.158327 4557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:49.159499 4557 hybrid_clock.cc:648] HybridClock initialized: now 1714682749159488 us; error 25 us; skew 500 ppm
I20240502 20:45:49.161424 4557 webserver.cc:460] Webserver started at http://127.30.32.67:37273/ using document root <none> and password file <none>
I20240502 20:45:49.161969 4557 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:49.162029 4557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:49.162196 4557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:49.163865 4557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "923cfab926e7443f8ffd7b21ad0a9d91"
format_stamp: "Formatted at 2024-05-02 20:45:49 on dist-test-slave-6336"
I20240502 20:45:49.164357 4557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "923cfab926e7443f8ffd7b21ad0a9d91"
format_stamp: "Formatted at 2024-05-02 20:45:49 on dist-test-slave-6336"
I20240502 20:45:49.167595 4557 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 20:45:49.169627 4571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:49.170557 4557 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:49.170660 4557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "923cfab926e7443f8ffd7b21ad0a9d91"
format_stamp: "Formatted at 2024-05-02 20:45:49 on dist-test-slave-6336"
I20240502 20:45:49.170737 4557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:49.191279 4557 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:49.191884 4557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:49.192075 4557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:49.192624 4557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:49.193604 4557 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:49.193655 4557 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:49.193714 4557 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:49.193753 4557 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:49.202211 4557 rpc_server.cc:307] RPC server started. Bound to: 127.30.32.67:42141
I20240502 20:45:49.202265 4684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.32.67:42141 every 8 connection(s)
I20240502 20:45:49.203204 4557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 20:45:49.203388 30849 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskolF7xA/build/debug/bin/kudu as pid 4557
I20240502 20:45:49.203498 30849 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 20:45:49.205924 30849 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskolF7xA/build/debug/bin/kudu
/tmp/dist-test-taskolF7xA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.32.68:0
--local_ip_for_outbound_sockets=127.30.32.68
--webserver_interface=127.30.32.68
--webserver_port=0
--tserver_master_addrs=127.30.32.126:40167
--builtin_ntp_servers=127.30.32.84:39781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 20:45:49.213152 4685 heartbeater.cc:347] Connected to a master server at 127.30.32.126:40167
I20240502 20:45:49.213371 4685 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:49.213971 4685 heartbeater.cc:511] Master 127.30.32.126:40167 requested a full tablet report, sending...
I20240502 20:45:49.215011 4234 ts_manager.cc:194] Registered new tserver with Master: 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:45:49.215795 4234 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.32.67:55029
W20240502 20:45:49.313669 4688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:49.313879 4688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:49.313922 4688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:49.316856 4688 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:49.316921 4688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:49.317049 4688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.32.68
I20240502 20:45:49.320771 4688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.32.84:39781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.32.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.30.32.68
--webserver_port=0
--tserver_master_addrs=127.30.32.126:40167
--never_fsync=true
--heap_profile_path=/tmp/kudu.4688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.32.68
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:49.321681 4688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:49.322773 4688 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:49.331339 4696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:49.332021 4693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:49.332608 4694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:49.333314 4688 server_base.cc:1034] running on GCE node
I20240502 20:45:49.333662 4688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:49.334162 4688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:49.335309 4688 hybrid_clock.cc:648] HybridClock initialized: now 1714682749335295 us; error 26 us; skew 500 ppm
I20240502 20:45:49.337102 4688 webserver.cc:460] Webserver started at http://127.30.32.68:42835/ using document root <none> and password file <none>
I20240502 20:45:49.337620 4688 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:49.337671 4688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:49.337826 4688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:49.339365 4688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "ed47c023452a497192f35d2f617bf9bd"
format_stamp: "Formatted at 2024-05-02 20:45:49 on dist-test-slave-6336"
I20240502 20:45:49.339778 4688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "ed47c023452a497192f35d2f617bf9bd"
format_stamp: "Formatted at 2024-05-02 20:45:49 on dist-test-slave-6336"
I20240502 20:45:49.342926 4688 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20240502 20:45:49.344933 4702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:49.345888 4688 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:49.345979 4688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "ed47c023452a497192f35d2f617bf9bd"
format_stamp: "Formatted at 2024-05-02 20:45:49 on dist-test-slave-6336"
I20240502 20:45:49.346050 4688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:49.371809 4688 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:49.372412 4688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:49.372547 4688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:49.373085 4688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:49.373986 4688 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:49.374033 4688 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:49.374073 4688 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:49.374095 4688 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:49.382079 4688 rpc_server.cc:307] RPC server started. Bound to: 127.30.32.68:38193
I20240502 20:45:49.382108 4815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.32.68:38193 every 8 connection(s)
I20240502 20:45:49.383018 4688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 20:45:49.391366 4816 heartbeater.cc:347] Connected to a master server at 127.30.32.126:40167
I20240502 20:45:49.391590 4816 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:49.392102 4816 heartbeater.cc:511] Master 127.30.32.126:40167 requested a full tablet report, sending...
I20240502 20:45:49.392622 30849 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskolF7xA/build/debug/bin/kudu as pid 4688
I20240502 20:45:49.392751 30849 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 20:45:49.393216 4234 ts_manager.cc:194] Registered new tserver with Master: ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:45:49.393847 4234 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.32.68:58705
I20240502 20:45:49.394817 30849 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskolF7xA/build/debug/bin/kudu
/tmp/dist-test-taskolF7xA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.32.69:0
--local_ip_for_outbound_sockets=127.30.32.69
--webserver_interface=127.30.32.69
--webserver_port=0
--tserver_master_addrs=127.30.32.126:40167
--builtin_ntp_servers=127.30.32.84:39781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 20:45:49.500301 4819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:49.500533 4819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:49.500579 4819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:49.503661 4819 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:49.503733 4819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:49.503830 4819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.32.69
I20240502 20:45:49.507464 4819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.32.84:39781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.32.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.30.32.69
--webserver_port=0
--tserver_master_addrs=127.30.32.126:40167
--never_fsync=true
--heap_profile_path=/tmp/kudu.4819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.32.69
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:49.508388 4819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:49.509527 4819 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:49.518462 4824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:49.518975 4827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:49.519552 4825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:49.520217 4819 server_base.cc:1034] running on GCE node
I20240502 20:45:49.520660 4819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:49.521306 4819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:49.522469 4819 hybrid_clock.cc:648] HybridClock initialized: now 1714682749522449 us; error 30 us; skew 500 ppm
I20240502 20:45:49.524551 4819 webserver.cc:460] Webserver started at http://127.30.32.69:40451/ using document root <none> and password file <none>
I20240502 20:45:49.525179 4819 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:49.525246 4819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:49.525444 4819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:49.527266 4819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "7845c397230d49358ba2fad9db53f97c"
format_stamp: "Formatted at 2024-05-02 20:45:49 on dist-test-slave-6336"
I20240502 20:45:49.527761 4819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "7845c397230d49358ba2fad9db53f97c"
format_stamp: "Formatted at 2024-05-02 20:45:49 on dist-test-slave-6336"
I20240502 20:45:49.531338 4819 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 20:45:49.533594 4833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:49.534622 4819 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 20:45:49.534744 4819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "7845c397230d49358ba2fad9db53f97c"
format_stamp: "Formatted at 2024-05-02 20:45:49 on dist-test-slave-6336"
I20240502 20:45:49.534838 4819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:49.556025 4819 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:49.556614 4819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:49.556773 4819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:49.557307 4819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:49.558141 4819 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:49.558198 4819 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:49.558257 4819 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:49.558285 4819 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:49.566622 4819 rpc_server.cc:307] RPC server started. Bound to: 127.30.32.69:42429
I20240502 20:45:49.566658 4946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.32.69:42429 every 8 connection(s)
I20240502 20:45:49.567508 4819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 20:45:49.570508 30849 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskolF7xA/build/debug/bin/kudu as pid 4819
I20240502 20:45:49.570612 30849 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 20:45:49.572672 30849 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskolF7xA/build/debug/bin/kudu
/tmp/dist-test-taskolF7xA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.32.70:0
--local_ip_for_outbound_sockets=127.30.32.70
--webserver_interface=127.30.32.70
--webserver_port=0
--tserver_master_addrs=127.30.32.126:40167
--builtin_ntp_servers=127.30.32.84:39781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 20:45:49.577339 4947 heartbeater.cc:347] Connected to a master server at 127.30.32.126:40167
I20240502 20:45:49.577570 4947 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:49.578091 4947 heartbeater.cc:511] Master 127.30.32.126:40167 requested a full tablet report, sending...
I20240502 20:45:49.579051 4235 ts_manager.cc:194] Registered new tserver with Master: 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
I20240502 20:45:49.579599 4235 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.32.69:57017
W20240502 20:45:49.684070 4950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:49.684291 4950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:49.684345 4950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:49.687505 4950 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:49.687610 4950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:49.687742 4950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.32.70
I20240502 20:45:49.691687 4950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.32.84:39781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.32.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.30.32.70
--webserver_port=0
--tserver_master_addrs=127.30.32.126:40167
--never_fsync=true
--heap_profile_path=/tmp/kudu.4950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.32.70
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:49.692899 4950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:49.694177 4950 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:49.703682 4956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:49.704341 4958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:49.705003 4955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:49.705675 4950 server_base.cc:1034] running on GCE node
I20240502 20:45:49.706153 4950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:49.706801 4950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:49.707962 4950 hybrid_clock.cc:648] HybridClock initialized: now 1714682749707943 us; error 39 us; skew 500 ppm
I20240502 20:45:49.710111 4950 webserver.cc:460] Webserver started at http://127.30.32.70:42737/ using document root <none> and password file <none>
I20240502 20:45:49.710700 4950 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:49.710762 4950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:49.710943 4950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:49.712646 4950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "a4bf1df7fa6b4948be502c182bfbb833"
format_stamp: "Formatted at 2024-05-02 20:45:49 on dist-test-slave-6336"
I20240502 20:45:49.713181 4950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "a4bf1df7fa6b4948be502c182bfbb833"
format_stamp: "Formatted at 2024-05-02 20:45:49 on dist-test-slave-6336"
I20240502 20:45:49.716533 4950 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20240502 20:45:49.718881 4964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:49.719935 4950 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:49.720116 4950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "a4bf1df7fa6b4948be502c182bfbb833"
format_stamp: "Formatted at 2024-05-02 20:45:49 on dist-test-slave-6336"
I20240502 20:45:49.720230 4950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:49.741170 4950 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:49.741806 4950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:49.741968 4950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:49.742484 4950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:49.743371 4950 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:49.743420 4950 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:49.743467 4950 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:49.743499 4950 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:49.752275 4950 rpc_server.cc:307] RPC server started. Bound to: 127.30.32.70:41867
I20240502 20:45:49.752362 5077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.32.70:41867 every 8 connection(s)
I20240502 20:45:49.753326 4950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 20:45:49.759513 30849 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskolF7xA/build/debug/bin/kudu as pid 4950
I20240502 20:45:49.759649 30849 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 20:45:49.761703 5078 heartbeater.cc:347] Connected to a master server at 127.30.32.126:40167
I20240502 20:45:49.761801 30849 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskolF7xA/build/debug/bin/kudu
/tmp/dist-test-taskolF7xA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.32.71:0
--local_ip_for_outbound_sockets=127.30.32.71
--webserver_interface=127.30.32.71
--webserver_port=0
--tserver_master_addrs=127.30.32.126:40167
--builtin_ntp_servers=127.30.32.84:39781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 20:45:49.761925 5078 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:49.762459 5078 heartbeater.cc:511] Master 127.30.32.126:40167 requested a full tablet report, sending...
I20240502 20:45:49.763461 4235 ts_manager.cc:194] Registered new tserver with Master: a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867)
I20240502 20:45:49.764158 4235 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.32.70:53943
I20240502 20:45:49.854799 4423 heartbeater.cc:503] Master 127.30.32.126:40167 was elected leader, sending a full tablet report...
W20240502 20:45:49.872855 5081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:49.873116 5081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:49.873188 5081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:49.876323 5081 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:49.876402 5081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:49.876498 5081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.32.71
I20240502 20:45:49.880064 5081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.32.84:39781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.32.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.30.32.71
--webserver_port=0
--tserver_master_addrs=127.30.32.126:40167
--never_fsync=true
--heap_profile_path=/tmp/kudu.5081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.32.71
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:49.880964 5081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:49.882086 5081 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:49.890012 5086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:49.890820 5089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:49.891041 5087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:49.891839 5081 server_base.cc:1034] running on GCE node
I20240502 20:45:49.892199 5081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:49.892730 5081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:49.893882 5081 hybrid_clock.cc:648] HybridClock initialized: now 1714682749893875 us; error 44 us; skew 500 ppm
I20240502 20:45:49.895639 5081 webserver.cc:460] Webserver started at http://127.30.32.71:35103/ using document root <none> and password file <none>
I20240502 20:45:49.896178 5081 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:49.896241 5081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:49.896412 5081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:49.898046 5081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "a47e5c56f13843018ac584914a8d4707"
format_stamp: "Formatted at 2024-05-02 20:45:49 on dist-test-slave-6336"
I20240502 20:45:49.898514 5081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "a47e5c56f13843018ac584914a8d4707"
format_stamp: "Formatted at 2024-05-02 20:45:49 on dist-test-slave-6336"
I20240502 20:45:49.901772 5081 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 20:45:49.903851 5095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:49.904775 5081 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 20:45:49.904879 5081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "a47e5c56f13843018ac584914a8d4707"
format_stamp: "Formatted at 2024-05-02 20:45:49 on dist-test-slave-6336"
I20240502 20:45:49.904961 5081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:49.920591 5081 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:49.921294 5081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:49.921452 5081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:49.921969 5081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:49.922834 5081 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:49.922880 5081 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:49.922928 5081 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:49.922960 5081 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:49.931555 5081 rpc_server.cc:307] RPC server started. Bound to: 127.30.32.71:46841
I20240502 20:45:49.931608 5208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.32.71:46841 every 8 connection(s)
I20240502 20:45:49.932443 5081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 20:45:49.937985 30849 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskolF7xA/build/debug/bin/kudu as pid 5081
I20240502 20:45:49.938079 30849 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 20:45:49.940353 30849 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskolF7xA/build/debug/bin/kudu
/tmp/dist-test-taskolF7xA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.32.72:0
--local_ip_for_outbound_sockets=127.30.32.72
--webserver_interface=127.30.32.72
--webserver_port=0
--tserver_master_addrs=127.30.32.126:40167
--builtin_ntp_servers=127.30.32.84:39781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 20:45:49.940996 5209 heartbeater.cc:347] Connected to a master server at 127.30.32.126:40167
I20240502 20:45:49.941208 5209 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:49.941697 5209 heartbeater.cc:511] Master 127.30.32.126:40167 requested a full tablet report, sending...
I20240502 20:45:49.942595 4235 ts_manager.cc:194] Registered new tserver with Master: a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
I20240502 20:45:49.943113 4235 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.32.71:60491
I20240502 20:45:50.028205 4554 heartbeater.cc:503] Master 127.30.32.126:40167 was elected leader, sending a full tablet report...
W20240502 20:45:50.046907 5212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:50.047113 5212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:50.047179 5212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:50.050176 5212 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:50.050246 5212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:50.050334 5212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.32.72
I20240502 20:45:50.053867 5212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.32.84:39781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.32.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.30.32.72
--webserver_port=0
--tserver_master_addrs=127.30.32.126:40167
--never_fsync=true
--heap_profile_path=/tmp/kudu.5212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.32.72
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:50.054687 5212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:50.055765 5212 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:50.063279 5217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:50.063517 5220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:50.064342 5218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:50.065039 5212 server_base.cc:1034] running on GCE node
I20240502 20:45:50.065383 5212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:50.065869 5212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:50.067008 5212 hybrid_clock.cc:648] HybridClock initialized: now 1714682750066990 us; error 31 us; skew 500 ppm
I20240502 20:45:50.068729 5212 webserver.cc:460] Webserver started at http://127.30.32.72:38697/ using document root <none> and password file <none>
I20240502 20:45:50.069307 5212 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:50.069365 5212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:50.069546 5212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:50.071094 5212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "201d2b6485084e7dad2cbb722af94779"
format_stamp: "Formatted at 2024-05-02 20:45:50 on dist-test-slave-6336"
I20240502 20:45:50.071540 5212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "201d2b6485084e7dad2cbb722af94779"
format_stamp: "Formatted at 2024-05-02 20:45:50 on dist-test-slave-6336"
I20240502 20:45:50.074721 5212 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20240502 20:45:50.077379 5226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:50.078512 5212 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240502 20:45:50.078629 5212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "201d2b6485084e7dad2cbb722af94779"
format_stamp: "Formatted at 2024-05-02 20:45:50 on dist-test-slave-6336"
I20240502 20:45:50.078720 5212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:50.106205 5212 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:50.106791 5212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:50.106940 5212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:50.107446 5212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:50.108331 5212 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:50.108376 5212 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:50.108422 5212 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:50.108453 5212 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:50.116555 5212 rpc_server.cc:307] RPC server started. Bound to: 127.30.32.72:44643
I20240502 20:45:50.116585 5339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.32.72:44643 every 8 connection(s)
I20240502 20:45:50.117491 5212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 20:45:50.125501 5340 heartbeater.cc:347] Connected to a master server at 127.30.32.126:40167
I20240502 20:45:50.125705 5340 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:50.126134 30849 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskolF7xA/build/debug/bin/kudu as pid 5212
I20240502 20:45:50.126189 5340 heartbeater.cc:511] Master 127.30.32.126:40167 requested a full tablet report, sending...
I20240502 20:45:50.126232 30849 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 20:45:50.127281 4234 ts_manager.cc:194] Registered new tserver with Master: 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
I20240502 20:45:50.127862 4234 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.32.72:43183
I20240502 20:45:50.128396 30849 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 20:45:50.137224 30849 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 20:45:50.137296 30849 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskolF7xA/build/debug/bin/kudu with pid 4819
I20240502 20:45:50.142517 30849 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskolF7xA/build/debug/bin/kudu with pid 4950
I20240502 20:45:50.147513 30849 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskolF7xA/build/debug/bin/kudu with pid 5081
I20240502 20:45:50.152920 30849 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskolF7xA/build/debug/bin/kudu with pid 5212
I20240502 20:45:50.217722 4685 heartbeater.cc:503] Master 127.30.32.126:40167 was elected leader, sending a full tablet report...
I20240502 20:45:50.395810 4816 heartbeater.cc:503] Master 127.30.32.126:40167 was elected leader, sending a full tablet report...
I20240502 20:45:53.909770 4235 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41812:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:45:53.929562 4355 tablet_service.cc:1467] Processing CreateTablet for tablet 84a90777f631447d8ba3ffb720d370e2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9a093bb257b04f07a867d916725eb265]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:53.929562 4356 tablet_service.cc:1467] Processing CreateTablet for tablet 48c3fb2ce0fc41a4a12a757052beccb8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9a093bb257b04f07a867d916725eb265]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:53.930614 4355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84a90777f631447d8ba3ffb720d370e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:53.930968 4356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48c3fb2ce0fc41a4a12a757052beccb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:53.931033 4488 tablet_service.cc:1467] Processing CreateTablet for tablet 3cb28ecff16240afb413a1d0e51733b0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9a093bb257b04f07a867d916725eb265]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:53.931914 4488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cb28ecff16240afb413a1d0e51733b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:53.933265 4487 tablet_service.cc:1467] Processing CreateTablet for tablet 48c3fb2ce0fc41a4a12a757052beccb8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9a093bb257b04f07a867d916725eb265]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:53.933516 4487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48c3fb2ce0fc41a4a12a757052beccb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:53.934062 4357 tablet_service.cc:1467] Processing CreateTablet for tablet 3cb28ecff16240afb413a1d0e51733b0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9a093bb257b04f07a867d916725eb265]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:53.934293 4357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cb28ecff16240afb413a1d0e51733b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:53.935515 4750 tablet_service.cc:1467] Processing CreateTablet for tablet 48c3fb2ce0fc41a4a12a757052beccb8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9a093bb257b04f07a867d916725eb265]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:53.936517 4750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48c3fb2ce0fc41a4a12a757052beccb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:53.937155 4749 tablet_service.cc:1467] Processing CreateTablet for tablet 84a90777f631447d8ba3ffb720d370e2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9a093bb257b04f07a867d916725eb265]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:53.937425 4749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84a90777f631447d8ba3ffb720d370e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:53.938249 5360 tablet_bootstrap.cc:492] T 48c3fb2ce0fc41a4a12a757052beccb8 P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:45:53.940099 5360 tablet_bootstrap.cc:654] T 48c3fb2ce0fc41a4a12a757052beccb8 P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:53.940757 5360 log.cc:824] T 48c3fb2ce0fc41a4a12a757052beccb8 P c92e6e46782849479c8a0c6d82276fb2: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:53.942523 5360 tablet_bootstrap.cc:492] T 48c3fb2ce0fc41a4a12a757052beccb8 P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:45:53.942700 5360 ts_tablet_manager.cc:1367] T 48c3fb2ce0fc41a4a12a757052beccb8 P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 20:45:53.943233 4618 tablet_service.cc:1467] Processing CreateTablet for tablet 84a90777f631447d8ba3ffb720d370e2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9a093bb257b04f07a867d916725eb265]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:53.943236 4619 tablet_service.cc:1467] Processing CreateTablet for tablet 3cb28ecff16240afb413a1d0e51733b0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9a093bb257b04f07a867d916725eb265]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:53.944216 4618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84a90777f631447d8ba3ffb720d370e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:53.944602 4619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cb28ecff16240afb413a1d0e51733b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:53.945987 5362 tablet_bootstrap.cc:492] T 84a90777f631447d8ba3ffb720d370e2 P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:45:53.946170 5360 raft_consensus.cc:348] T 48c3fb2ce0fc41a4a12a757052beccb8 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:53.946375 5360 raft_consensus.cc:374] T 48c3fb2ce0fc41a4a12a757052beccb8 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:53.946429 5360 raft_consensus.cc:729] T 48c3fb2ce0fc41a4a12a757052beccb8 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:45:53.946923 5360 consensus_queue.cc:260] T 48c3fb2ce0fc41a4a12a757052beccb8 P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:53.947783 5360 ts_tablet_manager.cc:1398] T 48c3fb2ce0fc41a4a12a757052beccb8 P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240502 20:45:53.948081 5362 tablet_bootstrap.cc:654] T 84a90777f631447d8ba3ffb720d370e2 P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:53.948860 5362 log.cc:824] T 84a90777f631447d8ba3ffb720d370e2 P ed47c023452a497192f35d2f617bf9bd: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:53.949450 5360 tablet_bootstrap.cc:492] T 3cb28ecff16240afb413a1d0e51733b0 P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:45:53.950311 5360 tablet_bootstrap.cc:654] T 3cb28ecff16240afb413a1d0e51733b0 P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:53.950335 5366 tablet_bootstrap.cc:492] T 84a90777f631447d8ba3ffb720d370e2 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:45:53.951097 5364 tablet_bootstrap.cc:492] T 3cb28ecff16240afb413a1d0e51733b0 P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:45:53.952394 5366 tablet_bootstrap.cc:654] T 84a90777f631447d8ba3ffb720d370e2 P 923cfab926e7443f8ffd7b21ad0a9d91: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:53.953099 5364 tablet_bootstrap.cc:654] T 3cb28ecff16240afb413a1d0e51733b0 P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:53.953186 5366 log.cc:824] T 84a90777f631447d8ba3ffb720d370e2 P 923cfab926e7443f8ffd7b21ad0a9d91: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:53.953841 5364 log.cc:824] T 3cb28ecff16240afb413a1d0e51733b0 P c3d26eafd51547acb780d4aac5ebe7cc: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:53.954414 5360 tablet_bootstrap.cc:492] T 3cb28ecff16240afb413a1d0e51733b0 P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:45:53.954509 5360 ts_tablet_manager.cc:1367] T 3cb28ecff16240afb413a1d0e51733b0 P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 20:45:53.955081 5366 tablet_bootstrap.cc:492] T 84a90777f631447d8ba3ffb720d370e2 P 923cfab926e7443f8ffd7b21ad0a9d91: No bootstrap required, opened a new log
I20240502 20:45:53.955057 5360 raft_consensus.cc:348] T 3cb28ecff16240afb413a1d0e51733b0 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:53.955168 5360 raft_consensus.cc:374] T 3cb28ecff16240afb413a1d0e51733b0 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:53.955201 5360 raft_consensus.cc:729] T 3cb28ecff16240afb413a1d0e51733b0 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:45:53.955250 5366 ts_tablet_manager.cc:1367] T 84a90777f631447d8ba3ffb720d370e2 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 20:45:53.955335 5364 tablet_bootstrap.cc:492] T 3cb28ecff16240afb413a1d0e51733b0 P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:45:53.955328 5360 consensus_queue.cc:260] T 3cb28ecff16240afb413a1d0e51733b0 P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:53.955487 5364 ts_tablet_manager.cc:1367] T 3cb28ecff16240afb413a1d0e51733b0 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240502 20:45:53.955601 5360 ts_tablet_manager.cc:1398] T 3cb28ecff16240afb413a1d0e51733b0 P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:45:53.955780 5360 tablet_bootstrap.cc:492] T 84a90777f631447d8ba3ffb720d370e2 P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:45:53.956633 5360 tablet_bootstrap.cc:654] T 84a90777f631447d8ba3ffb720d370e2 P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:53.957141 5362 tablet_bootstrap.cc:492] T 84a90777f631447d8ba3ffb720d370e2 P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:45:53.957450 5362 ts_tablet_manager.cc:1367] T 84a90777f631447d8ba3ffb720d370e2 P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240502 20:45:53.959492 5364 raft_consensus.cc:348] T 3cb28ecff16240afb413a1d0e51733b0 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:53.959492 5366 raft_consensus.cc:348] T 84a90777f631447d8ba3ffb720d370e2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:53.959728 5364 raft_consensus.cc:374] T 3cb28ecff16240afb413a1d0e51733b0 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:53.959728 5366 raft_consensus.cc:374] T 84a90777f631447d8ba3ffb720d370e2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:53.959789 5364 raft_consensus.cc:729] T 3cb28ecff16240afb413a1d0e51733b0 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:45:53.959789 5366 raft_consensus.cc:729] T 84a90777f631447d8ba3ffb720d370e2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: FOLLOWER
I20240502 20:45:53.960294 5366 consensus_queue.cc:260] T 84a90777f631447d8ba3ffb720d370e2 P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:53.960294 5364 consensus_queue.cc:260] T 3cb28ecff16240afb413a1d0e51733b0 P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:53.961225 5364 ts_tablet_manager.cc:1398] T 3cb28ecff16240afb413a1d0e51733b0 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20240502 20:45:53.961225 5366 ts_tablet_manager.cc:1398] T 84a90777f631447d8ba3ffb720d370e2 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240502 20:45:53.961247 5362 raft_consensus.cc:348] T 84a90777f631447d8ba3ffb720d370e2 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:53.961453 5362 raft_consensus.cc:374] T 84a90777f631447d8ba3ffb720d370e2 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:53.961513 5362 raft_consensus.cc:729] T 84a90777f631447d8ba3ffb720d370e2 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:45:53.961596 5364 tablet_bootstrap.cc:492] T 48c3fb2ce0fc41a4a12a757052beccb8 P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:45:53.961992 5362 consensus_queue.cc:260] T 84a90777f631447d8ba3ffb720d370e2 P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:53.962666 5366 tablet_bootstrap.cc:492] T 3cb28ecff16240afb413a1d0e51733b0 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:45:53.962812 5364 tablet_bootstrap.cc:654] T 48c3fb2ce0fc41a4a12a757052beccb8 P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:53.963863 5366 tablet_bootstrap.cc:654] T 3cb28ecff16240afb413a1d0e51733b0 P 923cfab926e7443f8ffd7b21ad0a9d91: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:53.964028 5364 tablet_bootstrap.cc:492] T 48c3fb2ce0fc41a4a12a757052beccb8 P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:45:53.964071 5362 ts_tablet_manager.cc:1398] T 84a90777f631447d8ba3ffb720d370e2 P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240502 20:45:53.964123 5364 ts_tablet_manager.cc:1367] T 48c3fb2ce0fc41a4a12a757052beccb8 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:45:53.964414 5362 tablet_bootstrap.cc:492] T 48c3fb2ce0fc41a4a12a757052beccb8 P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:45:53.964708 5364 raft_consensus.cc:348] T 48c3fb2ce0fc41a4a12a757052beccb8 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:53.964800 5364 raft_consensus.cc:374] T 48c3fb2ce0fc41a4a12a757052beccb8 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:53.964833 5364 raft_consensus.cc:729] T 48c3fb2ce0fc41a4a12a757052beccb8 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:45:53.965001 5364 consensus_queue.cc:260] T 48c3fb2ce0fc41a4a12a757052beccb8 P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:53.965409 5364 ts_tablet_manager.cc:1398] T 48c3fb2ce0fc41a4a12a757052beccb8 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 20:45:53.966598 5360 tablet_bootstrap.cc:492] T 84a90777f631447d8ba3ffb720d370e2 P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:45:53.966710 5360 ts_tablet_manager.cc:1367] T 84a90777f631447d8ba3ffb720d370e2 P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240502 20:45:53.967010 5362 tablet_bootstrap.cc:654] T 48c3fb2ce0fc41a4a12a757052beccb8 P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:53.967195 5360 raft_consensus.cc:348] T 84a90777f631447d8ba3ffb720d370e2 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:53.967335 5360 raft_consensus.cc:374] T 84a90777f631447d8ba3ffb720d370e2 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:53.967368 5360 raft_consensus.cc:729] T 84a90777f631447d8ba3ffb720d370e2 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:45:53.967532 5360 consensus_queue.cc:260] T 84a90777f631447d8ba3ffb720d370e2 P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:53.967846 5360 ts_tablet_manager.cc:1398] T 84a90777f631447d8ba3ffb720d370e2 P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:53.968569 5366 tablet_bootstrap.cc:492] T 3cb28ecff16240afb413a1d0e51733b0 P 923cfab926e7443f8ffd7b21ad0a9d91: No bootstrap required, opened a new log
I20240502 20:45:53.968655 5366 ts_tablet_manager.cc:1367] T 3cb28ecff16240afb413a1d0e51733b0 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 20:45:53.969265 5366 raft_consensus.cc:348] T 3cb28ecff16240afb413a1d0e51733b0 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:53.969369 5366 raft_consensus.cc:374] T 3cb28ecff16240afb413a1d0e51733b0 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:53.969378 5362 tablet_bootstrap.cc:492] T 48c3fb2ce0fc41a4a12a757052beccb8 P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:45:53.969411 5366 raft_consensus.cc:729] T 3cb28ecff16240afb413a1d0e51733b0 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: FOLLOWER
I20240502 20:45:53.969455 5362 ts_tablet_manager.cc:1367] T 48c3fb2ce0fc41a4a12a757052beccb8 P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 20:45:53.969533 5366 consensus_queue.cc:260] T 3cb28ecff16240afb413a1d0e51733b0 P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:53.969827 5366 ts_tablet_manager.cc:1398] T 3cb28ecff16240afb413a1d0e51733b0 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:53.970002 5362 raft_consensus.cc:348] T 48c3fb2ce0fc41a4a12a757052beccb8 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:53.970106 5362 raft_consensus.cc:374] T 48c3fb2ce0fc41a4a12a757052beccb8 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:53.970139 5362 raft_consensus.cc:729] T 48c3fb2ce0fc41a4a12a757052beccb8 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:45:53.970255 5362 consensus_queue.cc:260] T 48c3fb2ce0fc41a4a12a757052beccb8 P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:53.970508 5362 ts_tablet_manager.cc:1398] T 48c3fb2ce0fc41a4a12a757052beccb8 P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:53.973575 5373 raft_consensus.cc:482] T 48c3fb2ce0fc41a4a12a757052beccb8 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:53.973695 5373 raft_consensus.cc:504] T 48c3fb2ce0fc41a4a12a757052beccb8 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:53.974665 5373 leader_election.cc:288] T 48c3fb2ce0fc41a4a12a757052beccb8 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097), c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:45:53.979138 4377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c3fb2ce0fc41a4a12a757052beccb8" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2" is_pre_election: true
I20240502 20:45:53.979142 4508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c3fb2ce0fc41a4a12a757052beccb8" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" is_pre_election: true
I20240502 20:45:53.979405 4377 raft_consensus.cc:2454] T 48c3fb2ce0fc41a4a12a757052beccb8 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 0.
I20240502 20:45:53.979408 4508 raft_consensus.cc:2454] T 48c3fb2ce0fc41a4a12a757052beccb8 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 0.
I20240502 20:45:53.979802 4706 leader_election.cc:302] T 48c3fb2ce0fc41a4a12a757052beccb8 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:45:53.980163 5373 raft_consensus.cc:2781] T 48c3fb2ce0fc41a4a12a757052beccb8 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:53.980222 5373 raft_consensus.cc:482] T 48c3fb2ce0fc41a4a12a757052beccb8 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:53.980254 5373 raft_consensus.cc:3037] T 48c3fb2ce0fc41a4a12a757052beccb8 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:53.981158 5373 raft_consensus.cc:504] T 48c3fb2ce0fc41a4a12a757052beccb8 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:53.981487 5373 leader_election.cc:288] T 48c3fb2ce0fc41a4a12a757052beccb8 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 election: Requested vote from peers c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097), c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:45:53.981794 4508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c3fb2ce0fc41a4a12a757052beccb8" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc"
I20240502 20:45:53.981828 4377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c3fb2ce0fc41a4a12a757052beccb8" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2"
I20240502 20:45:53.981925 4377 raft_consensus.cc:3037] T 48c3fb2ce0fc41a4a12a757052beccb8 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:53.981932 4508 raft_consensus.cc:3037] T 48c3fb2ce0fc41a4a12a757052beccb8 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:53.983112 4508 raft_consensus.cc:2454] T 48c3fb2ce0fc41a4a12a757052beccb8 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 1.
I20240502 20:45:53.983426 4705 leader_election.cc:302] T 48c3fb2ce0fc41a4a12a757052beccb8 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3d26eafd51547acb780d4aac5ebe7cc, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:45:53.983438 4377 raft_consensus.cc:2454] T 48c3fb2ce0fc41a4a12a757052beccb8 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 1.
I20240502 20:45:53.983731 5373 raft_consensus.cc:2781] T 48c3fb2ce0fc41a4a12a757052beccb8 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:53.983954 5373 raft_consensus.cc:686] T 48c3fb2ce0fc41a4a12a757052beccb8 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Becoming Leader. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Running, Role: LEADER
I20240502 20:45:53.984248 5373 consensus_queue.cc:237] T 48c3fb2ce0fc41a4a12a757052beccb8 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:53.987337 4234 catalog_manager.cc:5484] T 48c3fb2ce0fc41a4a12a757052beccb8 P ed47c023452a497192f35d2f617bf9bd reported cstate change: term changed from 0 to 1, leader changed from <none> to ed47c023452a497192f35d2f617bf9bd (127.30.32.68). New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:54.018656 5371 raft_consensus.cc:482] T 84a90777f631447d8ba3ffb720d370e2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:54.018913 5371 raft_consensus.cc:504] T 84a90777f631447d8ba3ffb720d370e2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:54.020069 5371 leader_election.cc:288] T 84a90777f631447d8ba3ffb720d370e2 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193), c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
W20240502 20:45:54.020457 4555 tablet.cc:2369] T 3cb28ecff16240afb413a1d0e51733b0 P c3d26eafd51547acb780d4aac5ebe7cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 20:45:54.024639 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84a90777f631447d8ba3ffb720d370e2" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd" is_pre_election: true
I20240502 20:45:54.024724 4377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84a90777f631447d8ba3ffb720d370e2" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2" is_pre_election: true
I20240502 20:45:54.024854 4377 raft_consensus.cc:2454] T 84a90777f631447d8ba3ffb720d370e2 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 0.
I20240502 20:45:54.024871 4770 raft_consensus.cc:2454] T 84a90777f631447d8ba3ffb720d370e2 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 0.
I20240502 20:45:54.025221 4575 leader_election.cc:302] T 84a90777f631447d8ba3ffb720d370e2 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:45:54.025463 5371 raft_consensus.cc:2781] T 84a90777f631447d8ba3ffb720d370e2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:54.025532 5371 raft_consensus.cc:482] T 84a90777f631447d8ba3ffb720d370e2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:54.025563 5371 raft_consensus.cc:3037] T 84a90777f631447d8ba3ffb720d370e2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:54.026507 5371 raft_consensus.cc:504] T 84a90777f631447d8ba3ffb720d370e2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:54.026844 5371 leader_election.cc:288] T 84a90777f631447d8ba3ffb720d370e2 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 election: Requested vote from peers ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193), c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:45:54.027165 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84a90777f631447d8ba3ffb720d370e2" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd"
I20240502 20:45:54.027277 4770 raft_consensus.cc:3037] T 84a90777f631447d8ba3ffb720d370e2 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:54.027225 4377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84a90777f631447d8ba3ffb720d370e2" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2"
I20240502 20:45:54.027383 4377 raft_consensus.cc:3037] T 84a90777f631447d8ba3ffb720d370e2 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:54.028051 4770 raft_consensus.cc:2454] T 84a90777f631447d8ba3ffb720d370e2 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 1.
I20240502 20:45:54.028148 4377 raft_consensus.cc:2454] T 84a90777f631447d8ba3ffb720d370e2 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 1.
I20240502 20:45:54.028339 4575 leader_election.cc:302] T 84a90777f631447d8ba3ffb720d370e2 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:45:54.028504 5371 raft_consensus.cc:2781] T 84a90777f631447d8ba3ffb720d370e2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:54.028712 5371 raft_consensus.cc:686] T 84a90777f631447d8ba3ffb720d370e2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Becoming Leader. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Running, Role: LEADER
I20240502 20:45:54.029058 5371 consensus_queue.cc:237] T 84a90777f631447d8ba3ffb720d370e2 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:54.031978 4234 catalog_manager.cc:5484] T 84a90777f631447d8ba3ffb720d370e2 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67). New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: UNKNOWN } } }
W20240502 20:45:54.094722 4424 tablet.cc:2369] T 3cb28ecff16240afb413a1d0e51733b0 P c92e6e46782849479c8a0c6d82276fb2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 20:45:54.128718 5371 raft_consensus.cc:482] T 3cb28ecff16240afb413a1d0e51733b0 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:54.128870 5371 raft_consensus.cc:504] T 3cb28ecff16240afb413a1d0e51733b0 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:54.129865 5371 leader_election.cc:288] T 3cb28ecff16240afb413a1d0e51733b0 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827), c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097)
I20240502 20:45:54.129977 4377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cb28ecff16240afb413a1d0e51733b0" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2" is_pre_election: true
I20240502 20:45:54.130149 4377 raft_consensus.cc:2454] T 3cb28ecff16240afb413a1d0e51733b0 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 0.
I20240502 20:45:54.130613 4575 leader_election.cc:302] T 3cb28ecff16240afb413a1d0e51733b0 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:45:54.130764 5371 raft_consensus.cc:2781] T 3cb28ecff16240afb413a1d0e51733b0 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:54.130843 5371 raft_consensus.cc:482] T 3cb28ecff16240afb413a1d0e51733b0 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:54.130895 5371 raft_consensus.cc:3037] T 3cb28ecff16240afb413a1d0e51733b0 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:54.131778 5371 raft_consensus.cc:504] T 3cb28ecff16240afb413a1d0e51733b0 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:54.132148 5371 leader_election.cc:288] T 3cb28ecff16240afb413a1d0e51733b0 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 election: Requested vote from peers c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827), c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097)
I20240502 20:45:54.132513 4377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cb28ecff16240afb413a1d0e51733b0" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2"
I20240502 20:45:54.132659 4377 raft_consensus.cc:3037] T 3cb28ecff16240afb413a1d0e51733b0 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:54.133747 4377 raft_consensus.cc:2454] T 3cb28ecff16240afb413a1d0e51733b0 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 1.
I20240502 20:45:54.133919 4508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cb28ecff16240afb413a1d0e51733b0" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" is_pre_election: true
I20240502 20:45:54.133931 4507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cb28ecff16240afb413a1d0e51733b0" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc"
I20240502 20:45:54.134073 4508 raft_consensus.cc:2454] T 3cb28ecff16240afb413a1d0e51733b0 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 0.
I20240502 20:45:54.134474 4575 leader_election.cc:302] T 3cb28ecff16240afb413a1d0e51733b0 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:45:54.134675 5371 raft_consensus.cc:2781] T 3cb28ecff16240afb413a1d0e51733b0 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:54.134757 5371 raft_consensus.cc:686] T 3cb28ecff16240afb413a1d0e51733b0 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Becoming Leader. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Running, Role: LEADER
I20240502 20:45:54.134923 5371 consensus_queue.cc:237] T 3cb28ecff16240afb413a1d0e51733b0 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:54.136646 4234 catalog_manager.cc:5484] T 3cb28ecff16240afb413a1d0e51733b0 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67). New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: UNKNOWN } } }
W20240502 20:45:54.137308 4817 tablet.cc:2369] T 48c3fb2ce0fc41a4a12a757052beccb8 P ed47c023452a497192f35d2f617bf9bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 20:45:54.208238 4686 tablet.cc:2369] T 3cb28ecff16240afb413a1d0e51733b0 P 923cfab926e7443f8ffd7b21ad0a9d91: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 20:45:54.273172 5387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:54.273469 5387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:54.276650 5387 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240502 20:45:54.370738 4507 raft_consensus.cc:1261] T 48c3fb2ce0fc41a4a12a757052beccb8 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:54.370795 4377 raft_consensus.cc:1261] T 48c3fb2ce0fc41a4a12a757052beccb8 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:54.371462 5380 consensus_queue.cc:1035] T 48c3fb2ce0fc41a4a12a757052beccb8 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:54.371682 5373 consensus_queue.cc:1035] T 48c3fb2ce0fc41a4a12a757052beccb8 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:54.373291 4376 raft_consensus.cc:1261] T 3cb28ecff16240afb413a1d0e51733b0 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:54.373329 4507 raft_consensus.cc:3037] T 3cb28ecff16240afb413a1d0e51733b0 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:54.374524 4507 raft_consensus.cc:1261] T 3cb28ecff16240afb413a1d0e51733b0 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:54.377569 5384 consensus_queue.cc:1035] T 3cb28ecff16240afb413a1d0e51733b0 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:54.378104 5384 consensus_queue.cc:1035] T 3cb28ecff16240afb413a1d0e51733b0 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:54.380765 4375 raft_consensus.cc:1261] T 84a90777f631447d8ba3ffb720d370e2 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:54.381888 4770 raft_consensus.cc:1261] T 84a90777f631447d8ba3ffb720d370e2 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:54.382253 5371 consensus_queue.cc:1035] T 84a90777f631447d8ba3ffb720d370e2 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:54.383329 5384 consensus_queue.cc:1035] T 84a90777f631447d8ba3ffb720d370e2 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:54.387545 5416 mvcc.cc:205] Tried to move back new op lower bound from 7023340561894981632 to 7023340560321945600. Current Snapshot: MvccSnapshot[applied={T|T < 7023340561894981632}]
I20240502 20:45:54.391708 5419 mvcc.cc:205] Tried to move back new op lower bound from 7023340561894981632 to 7023340560321945600. Current Snapshot: MvccSnapshot[applied={T|T < 7023340561894981632}]
I20240502 20:45:54.394727 5417 mvcc.cc:205] Tried to move back new op lower bound from 7023340561923510272 to 7023340560937934848. Current Snapshot: MvccSnapshot[applied={T|T < 7023340561900511232}]
INSERT report
rows total: 2000
time total: 122.554 ms
time per row: 0.061277 ms
I20240502 20:45:54.448349 4228 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41812:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:45:54.454128 4619 tablet_service.cc:1467] Processing CreateTablet for tablet a2e81d2c63cd4e4cac5086506b5c4a12 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3a26d60c6de746e891d970c2ce9c55cc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:54.454347 4488 tablet_service.cc:1467] Processing CreateTablet for tablet a2e81d2c63cd4e4cac5086506b5c4a12 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3a26d60c6de746e891d970c2ce9c55cc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:54.454468 4619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2e81d2c63cd4e4cac5086506b5c4a12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:54.454607 4488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2e81d2c63cd4e4cac5086506b5c4a12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:54.455276 4749 tablet_service.cc:1467] Processing CreateTablet for tablet 28da7347f982468281570ffa7503a854 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3a26d60c6de746e891d970c2ce9c55cc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:54.455278 4355 tablet_service.cc:1467] Processing CreateTablet for tablet 76b6dee18d224e879bf0d3b6bbb2b9b5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3a26d60c6de746e891d970c2ce9c55cc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:54.455543 4355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76b6dee18d224e879bf0d3b6bbb2b9b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:54.455543 4749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28da7347f982468281570ffa7503a854. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:54.456686 5364 tablet_bootstrap.cc:492] T a2e81d2c63cd4e4cac5086506b5c4a12 P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:45:54.456861 4750 tablet_service.cc:1467] Processing CreateTablet for tablet a2e81d2c63cd4e4cac5086506b5c4a12 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3a26d60c6de746e891d970c2ce9c55cc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:54.457124 5366 tablet_bootstrap.cc:492] T a2e81d2c63cd4e4cac5086506b5c4a12 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:45:54.457566 5364 tablet_bootstrap.cc:654] T a2e81d2c63cd4e4cac5086506b5c4a12 P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:54.457574 5360 tablet_bootstrap.cc:492] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:45:54.457955 5366 tablet_bootstrap.cc:654] T a2e81d2c63cd4e4cac5086506b5c4a12 P 923cfab926e7443f8ffd7b21ad0a9d91: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:54.458434 5360 tablet_bootstrap.cc:654] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:54.458501 4618 tablet_service.cc:1467] Processing CreateTablet for tablet 76b6dee18d224e879bf0d3b6bbb2b9b5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3a26d60c6de746e891d970c2ce9c55cc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:54.458755 4618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76b6dee18d224e879bf0d3b6bbb2b9b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:54.458978 5366 tablet_bootstrap.cc:492] T a2e81d2c63cd4e4cac5086506b5c4a12 P 923cfab926e7443f8ffd7b21ad0a9d91: No bootstrap required, opened a new log
I20240502 20:45:54.459049 5366 ts_tablet_manager.cc:1367] T a2e81d2c63cd4e4cac5086506b5c4a12 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:54.459425 5366 raft_consensus.cc:348] T a2e81d2c63cd4e4cac5086506b5c4a12 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:54.459517 5366 raft_consensus.cc:374] T a2e81d2c63cd4e4cac5086506b5c4a12 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:54.459551 5366 raft_consensus.cc:729] T a2e81d2c63cd4e4cac5086506b5c4a12 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: FOLLOWER
I20240502 20:45:54.459692 5366 consensus_queue.cc:260] T a2e81d2c63cd4e4cac5086506b5c4a12 P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:54.459947 5366 ts_tablet_manager.cc:1398] T a2e81d2c63cd4e4cac5086506b5c4a12 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:45:54.460374 5366 tablet_bootstrap.cc:492] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:45:54.460666 5360 tablet_bootstrap.cc:492] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:45:54.460749 4750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2e81d2c63cd4e4cac5086506b5c4a12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:54.460762 5360 ts_tablet_manager.cc:1367] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:45:54.461200 5366 tablet_bootstrap.cc:654] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:54.461532 4488 tablet_service.cc:1467] Processing CreateTablet for tablet 28da7347f982468281570ffa7503a854 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3a26d60c6de746e891d970c2ce9c55cc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:54.461766 4488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28da7347f982468281570ffa7503a854. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:54.462594 5362 tablet_bootstrap.cc:492] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:45:54.463413 5362 tablet_bootstrap.cc:654] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:54.463430 4619 tablet_service.cc:1467] Processing CreateTablet for tablet 28da7347f982468281570ffa7503a854 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3a26d60c6de746e891d970c2ce9c55cc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:54.463668 4619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28da7347f982468281570ffa7503a854. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:54.463848 4487 tablet_service.cc:1467] Processing CreateTablet for tablet 76b6dee18d224e879bf0d3b6bbb2b9b5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3a26d60c6de746e891d970c2ce9c55cc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:54.464066 4487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76b6dee18d224e879bf0d3b6bbb2b9b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:54.464534 5362 tablet_bootstrap.cc:492] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:45:54.464612 5362 ts_tablet_manager.cc:1367] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:45:54.465049 5362 raft_consensus.cc:348] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:54.465148 5362 raft_consensus.cc:374] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:54.465184 5362 raft_consensus.cc:729] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:45:54.465308 5362 consensus_queue.cc:260] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:54.465576 5362 ts_tablet_manager.cc:1398] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:54.465735 5362 tablet_bootstrap.cc:492] T 28da7347f982468281570ffa7503a854 P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:45:54.466395 5364 tablet_bootstrap.cc:492] T a2e81d2c63cd4e4cac5086506b5c4a12 P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:45:54.466473 5364 ts_tablet_manager.cc:1367] T a2e81d2c63cd4e4cac5086506b5c4a12 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240502 20:45:54.466657 5360 raft_consensus.cc:348] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:54.466770 5360 raft_consensus.cc:374] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:54.466806 5360 raft_consensus.cc:729] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:45:54.466804 5364 raft_consensus.cc:348] T a2e81d2c63cd4e4cac5086506b5c4a12 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:54.466877 5364 raft_consensus.cc:374] T a2e81d2c63cd4e4cac5086506b5c4a12 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:54.466897 5364 raft_consensus.cc:729] T a2e81d2c63cd4e4cac5086506b5c4a12 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:45:54.466933 5362 tablet_bootstrap.cc:654] T 28da7347f982468281570ffa7503a854 P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:54.466955 5360 consensus_queue.cc:260] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:54.467010 5364 consensus_queue.cc:260] T a2e81d2c63cd4e4cac5086506b5c4a12 P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:54.467270 5360 ts_tablet_manager.cc:1398] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240502 20:45:54.467273 5364 ts_tablet_manager.cc:1398] T a2e81d2c63cd4e4cac5086506b5c4a12 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:54.467427 5364 tablet_bootstrap.cc:492] T 28da7347f982468281570ffa7503a854 P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:45:54.468014 5366 tablet_bootstrap.cc:492] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91: No bootstrap required, opened a new log
I20240502 20:45:54.468096 5366 ts_tablet_manager.cc:1367] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240502 20:45:54.468276 5364 tablet_bootstrap.cc:654] T 28da7347f982468281570ffa7503a854 P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:54.468545 5366 raft_consensus.cc:348] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:54.468641 5366 raft_consensus.cc:374] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:54.468675 5366 raft_consensus.cc:729] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: FOLLOWER
I20240502 20:45:54.468782 5366 consensus_queue.cc:260] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:54.469099 5366 ts_tablet_manager.cc:1398] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:54.469265 5366 tablet_bootstrap.cc:492] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:45:54.469329 5380 raft_consensus.cc:482] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:54.469393 5380 raft_consensus.cc:504] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:54.469823 5380 leader_election.cc:288] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141), c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097)
I20240502 20:45:54.470058 5366 tablet_bootstrap.cc:654] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:54.470135 5364 tablet_bootstrap.cc:492] T 28da7347f982468281570ffa7503a854 P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:45:54.470206 5364 ts_tablet_manager.cc:1367] T 28da7347f982468281570ffa7503a854 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 20:45:54.470640 5364 raft_consensus.cc:348] T 28da7347f982468281570ffa7503a854 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:54.470757 5364 raft_consensus.cc:374] T 28da7347f982468281570ffa7503a854 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:54.470798 5364 raft_consensus.cc:729] T 28da7347f982468281570ffa7503a854 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:45:54.470909 5364 consensus_queue.cc:260] T 28da7347f982468281570ffa7503a854 P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:54.471169 5364 ts_tablet_manager.cc:1398] T 28da7347f982468281570ffa7503a854 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:54.471331 5364 tablet_bootstrap.cc:492] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:45:54.471755 5366 tablet_bootstrap.cc:492] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91: No bootstrap required, opened a new log
I20240502 20:45:54.471897 5366 ts_tablet_manager.cc:1367] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:45:54.472183 5364 tablet_bootstrap.cc:654] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:54.472426 5366 raft_consensus.cc:348] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:54.472623 5366 raft_consensus.cc:374] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:54.472704 5366 raft_consensus.cc:729] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: FOLLOWER
I20240502 20:45:54.472909 5366 consensus_queue.cc:260] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:54.473297 5366 ts_tablet_manager.cc:1398] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:54.470207 4508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2e81d2c63cd4e4cac5086506b5c4a12" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" is_pre_election: true
I20240502 20:45:54.474035 5362 tablet_bootstrap.cc:492] T 28da7347f982468281570ffa7503a854 P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:45:54.474045 4508 raft_consensus.cc:2454] T a2e81d2c63cd4e4cac5086506b5c4a12 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 0.
I20240502 20:45:54.474105 5362 ts_tablet_manager.cc:1367] T 28da7347f982468281570ffa7503a854 P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 20:45:54.474351 4705 leader_election.cc:302] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3d26eafd51547acb780d4aac5ebe7cc, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:45:54.474511 5380 raft_consensus.cc:2781] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:54.474582 5380 raft_consensus.cc:482] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:54.474644 5380 raft_consensus.cc:3037] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:54.474522 5362 raft_consensus.cc:348] T 28da7347f982468281570ffa7503a854 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:54.475251 5362 raft_consensus.cc:374] T 28da7347f982468281570ffa7503a854 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:54.475322 5362 raft_consensus.cc:729] T 28da7347f982468281570ffa7503a854 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:45:54.475472 5362 consensus_queue.cc:260] T 28da7347f982468281570ffa7503a854 P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:54.475637 5380 raft_consensus.cc:504] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:54.476401 5362 ts_tablet_manager.cc:1398] T 28da7347f982468281570ffa7503a854 P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:45:54.476580 5380 leader_election.cc:288] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 election: Requested vote from peers 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141), c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097)
I20240502 20:45:54.476955 4508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2e81d2c63cd4e4cac5086506b5c4a12" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc"
I20240502 20:45:54.477077 4508 raft_consensus.cc:3037] T a2e81d2c63cd4e4cac5086506b5c4a12 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:54.477945 5364 tablet_bootstrap.cc:492] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:45:54.477967 4508 raft_consensus.cc:2454] T a2e81d2c63cd4e4cac5086506b5c4a12 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 1.
I20240502 20:45:54.478032 5364 ts_tablet_manager.cc:1367] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 20:45:54.478232 4705 leader_election.cc:302] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3d26eafd51547acb780d4aac5ebe7cc, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:45:54.478343 4639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2e81d2c63cd4e4cac5086506b5c4a12" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" is_pre_election: true
I20240502 20:45:54.478386 5380 raft_consensus.cc:2781] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:54.478471 5380 raft_consensus.cc:686] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Becoming Leader. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Running, Role: LEADER
I20240502 20:45:54.478440 5364 raft_consensus.cc:348] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:54.478519 5364 raft_consensus.cc:374] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:54.478530 4639 raft_consensus.cc:2454] T a2e81d2c63cd4e4cac5086506b5c4a12 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 0.
I20240502 20:45:54.478552 5364 raft_consensus.cc:729] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:45:54.478598 5380 consensus_queue.cc:237] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:54.478786 4638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2e81d2c63cd4e4cac5086506b5c4a12" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91"
I20240502 20:45:54.478899 4638 raft_consensus.cc:3037] T a2e81d2c63cd4e4cac5086506b5c4a12 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:54.478909 5364 consensus_queue.cc:260] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:54.479187 5364 ts_tablet_manager.cc:1398] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:54.479908 4638 raft_consensus.cc:2454] T a2e81d2c63cd4e4cac5086506b5c4a12 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 1.
I20240502 20:45:54.481241 4228 catalog_manager.cc:5484] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd reported cstate change: term changed from 0 to 1, leader changed from <none> to ed47c023452a497192f35d2f617bf9bd (127.30.32.68). New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } }
I20240502 20:45:54.502749 5384 raft_consensus.cc:482] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:54.502920 5384 raft_consensus.cc:504] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:54.503356 5384 leader_election.cc:288] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097), c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:45:54.503757 4377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76b6dee18d224e879bf0d3b6bbb2b9b5" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2" is_pre_election: true
I20240502 20:45:54.503758 4508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76b6dee18d224e879bf0d3b6bbb2b9b5" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" is_pre_election: true
I20240502 20:45:54.503888 4508 raft_consensus.cc:2454] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 0.
I20240502 20:45:54.503893 4377 raft_consensus.cc:2454] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 0.
I20240502 20:45:54.504207 4574 leader_election.cc:302] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, c3d26eafd51547acb780d4aac5ebe7cc; no voters:
I20240502 20:45:54.504379 5384 raft_consensus.cc:2781] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:54.504449 5384 raft_consensus.cc:482] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:54.504518 5384 raft_consensus.cc:3037] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:54.505594 5384 raft_consensus.cc:504] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:54.505882 5384 leader_election.cc:288] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 election: Requested vote from peers c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097), c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:45:54.506194 4508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76b6dee18d224e879bf0d3b6bbb2b9b5" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc"
I20240502 20:45:54.506227 4377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76b6dee18d224e879bf0d3b6bbb2b9b5" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2"
I20240502 20:45:54.506304 4508 raft_consensus.cc:3037] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:54.506336 4377 raft_consensus.cc:3037] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:54.506999 4508 raft_consensus.cc:2454] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 1.
I20240502 20:45:54.507014 4377 raft_consensus.cc:2454] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 1.
I20240502 20:45:54.507265 4575 leader_election.cc:302] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:45:54.507429 5384 raft_consensus.cc:2781] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:54.507514 5384 raft_consensus.cc:686] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Becoming Leader. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Running, Role: LEADER
I20240502 20:45:54.507671 5384 consensus_queue.cc:237] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:54.509209 4228 catalog_manager.cc:5484] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67). New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:54.620617 5384 raft_consensus.cc:482] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:54.620760 5384 raft_consensus.cc:504] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:54.621227 5384 leader_election.cc:288] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097), ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:45:54.621701 4508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28da7347f982468281570ffa7503a854" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" is_pre_election: true
I20240502 20:45:54.621739 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28da7347f982468281570ffa7503a854" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd" is_pre_election: true
I20240502 20:45:54.621876 4770 raft_consensus.cc:2454] T 28da7347f982468281570ffa7503a854 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 0.
I20240502 20:45:54.621876 4508 raft_consensus.cc:2454] T 28da7347f982468281570ffa7503a854 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 0.
I20240502 20:45:54.622184 4575 leader_election.cc:302] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:45:54.622382 5384 raft_consensus.cc:2781] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:54.622447 5384 raft_consensus.cc:482] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:54.622498 5384 raft_consensus.cc:3037] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:54.623327 5384 raft_consensus.cc:504] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:54.623688 5384 leader_election.cc:288] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 election: Requested vote from peers c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097), ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:45:54.624186 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28da7347f982468281570ffa7503a854" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd"
I20240502 20:45:54.624186 4508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28da7347f982468281570ffa7503a854" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc"
I20240502 20:45:54.624325 4770 raft_consensus.cc:3037] T 28da7347f982468281570ffa7503a854 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:54.624325 4508 raft_consensus.cc:3037] T 28da7347f982468281570ffa7503a854 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:54.625279 4770 raft_consensus.cc:2454] T 28da7347f982468281570ffa7503a854 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 1.
I20240502 20:45:54.625293 4508 raft_consensus.cc:2454] T 28da7347f982468281570ffa7503a854 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 1.
I20240502 20:45:54.625602 4574 leader_election.cc:302] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, c3d26eafd51547acb780d4aac5ebe7cc; no voters:
I20240502 20:45:54.625779 5384 raft_consensus.cc:2781] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:54.625871 5384 raft_consensus.cc:686] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Becoming Leader. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Running, Role: LEADER
I20240502 20:45:54.626010 5384 consensus_queue.cc:237] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:54.627672 4228 catalog_manager.cc:5484] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67). New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: UNKNOWN } } }
W20240502 20:45:54.759832 5460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:54.760107 5460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:54.763166 5460 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240502 20:45:54.842259 4377 raft_consensus.cc:1261] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:54.843042 4507 raft_consensus.cc:1261] T 28da7347f982468281570ffa7503a854 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:54.843286 4770 raft_consensus.cc:1261] T 28da7347f982468281570ffa7503a854 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:54.843492 5384 consensus_queue.cc:1035] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:54.844092 5371 consensus_queue.cc:1035] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:54.844391 5371 consensus_queue.cc:1035] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:54.845186 4508 raft_consensus.cc:1261] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:54.846549 4508 raft_consensus.cc:1261] T a2e81d2c63cd4e4cac5086506b5c4a12 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:54.846971 5380 consensus_queue.cc:1035] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:54.849882 4638 raft_consensus.cc:1261] T a2e81d2c63cd4e4cac5086506b5c4a12 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:54.850337 5384 consensus_queue.cc:1035] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:54.854759 5373 consensus_queue.cc:1035] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:54.873220 5420 mvcc.cc:205] Tried to move back new op lower bound from 7023340563829211136 to 7023340562464886784. Current Snapshot: MvccSnapshot[applied={T|T < 7023340563829211136}]
INSERT report
rows total: 2000
time total: 108.772 ms
time per row: 0.0543861 ms
I20240502 20:45:54.915341 4234 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41812:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:45:54.921321 4355 tablet_service.cc:1467] Processing CreateTablet for tablet 371e10ffd8e64a33bbd950a140c7c514 (DEFAULT_TABLE table=rebalance_test_table_2 [id=59de2d64647446ab856b853639a3b1ca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:54.921469 4487 tablet_service.cc:1467] Processing CreateTablet for tablet 371e10ffd8e64a33bbd950a140c7c514 (DEFAULT_TABLE table=rebalance_test_table_2 [id=59de2d64647446ab856b853639a3b1ca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:54.921561 4749 tablet_service.cc:1467] Processing CreateTablet for tablet 371e10ffd8e64a33bbd950a140c7c514 (DEFAULT_TABLE table=rebalance_test_table_2 [id=59de2d64647446ab856b853639a3b1ca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:54.921682 4355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 371e10ffd8e64a33bbd950a140c7c514. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:54.921772 4487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 371e10ffd8e64a33bbd950a140c7c514. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:54.921808 4749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 371e10ffd8e64a33bbd950a140c7c514. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:54.922921 4488 tablet_service.cc:1467] Processing CreateTablet for tablet bdac13ec12c24d28abcca7f1e451451d (DEFAULT_TABLE table=rebalance_test_table_2 [id=59de2d64647446ab856b853639a3b1ca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:54.923166 4488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdac13ec12c24d28abcca7f1e451451d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:54.924052 4619 tablet_service.cc:1467] Processing CreateTablet for tablet 85e3a77b9daf4f3f93a10b223b168064 (DEFAULT_TABLE table=rebalance_test_table_2 [id=59de2d64647446ab856b853639a3b1ca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:54.924295 4619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85e3a77b9daf4f3f93a10b223b168064. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:54.924248 4355 tablet_service.cc:1467] Processing CreateTablet for tablet 85e3a77b9daf4f3f93a10b223b168064 (DEFAULT_TABLE table=rebalance_test_table_2 [id=59de2d64647446ab856b853639a3b1ca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:54.924451 4355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85e3a77b9daf4f3f93a10b223b168064. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:54.926064 5366 tablet_bootstrap.cc:492] T 85e3a77b9daf4f3f93a10b223b168064 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:45:54.926158 5362 tablet_bootstrap.cc:492] T 371e10ffd8e64a33bbd950a140c7c514 P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:45:54.927001 5362 tablet_bootstrap.cc:654] T 371e10ffd8e64a33bbd950a140c7c514 P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:54.927018 5366 tablet_bootstrap.cc:654] T 85e3a77b9daf4f3f93a10b223b168064 P 923cfab926e7443f8ffd7b21ad0a9d91: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:54.927614 5364 tablet_bootstrap.cc:492] T 371e10ffd8e64a33bbd950a140c7c514 P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:45:54.928189 5362 tablet_bootstrap.cc:492] T 371e10ffd8e64a33bbd950a140c7c514 P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:45:54.928282 5362 ts_tablet_manager.cc:1367] T 371e10ffd8e64a33bbd950a140c7c514 P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:45:54.928316 5366 tablet_bootstrap.cc:492] T 85e3a77b9daf4f3f93a10b223b168064 P 923cfab926e7443f8ffd7b21ad0a9d91: No bootstrap required, opened a new log
I20240502 20:45:54.928409 5366 ts_tablet_manager.cc:1367] T 85e3a77b9daf4f3f93a10b223b168064 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:54.928490 5364 tablet_bootstrap.cc:654] T 371e10ffd8e64a33bbd950a140c7c514 P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:54.928809 5366 raft_consensus.cc:348] T 85e3a77b9daf4f3f93a10b223b168064 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:54.928804 5362 raft_consensus.cc:348] T 371e10ffd8e64a33bbd950a140c7c514 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:54.928910 5366 raft_consensus.cc:374] T 85e3a77b9daf4f3f93a10b223b168064 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:54.928921 5362 raft_consensus.cc:374] T 371e10ffd8e64a33bbd950a140c7c514 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:54.928944 5366 raft_consensus.cc:729] T 85e3a77b9daf4f3f93a10b223b168064 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: FOLLOWER
I20240502 20:45:54.928998 5362 raft_consensus.cc:729] T 371e10ffd8e64a33bbd950a140c7c514 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:45:54.929049 5360 tablet_bootstrap.cc:492] T 371e10ffd8e64a33bbd950a140c7c514 P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:45:54.929103 5366 consensus_queue.cc:260] T 85e3a77b9daf4f3f93a10b223b168064 P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:54.929129 5362 consensus_queue.cc:260] T 371e10ffd8e64a33bbd950a140c7c514 P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:54.929381 5366 ts_tablet_manager.cc:1398] T 85e3a77b9daf4f3f93a10b223b168064 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:45:54.929404 5362 ts_tablet_manager.cc:1398] T 371e10ffd8e64a33bbd950a140c7c514 P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:54.929965 5360 tablet_bootstrap.cc:654] T 371e10ffd8e64a33bbd950a140c7c514 P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:54.930794 4357 tablet_service.cc:1467] Processing CreateTablet for tablet bdac13ec12c24d28abcca7f1e451451d (DEFAULT_TABLE table=rebalance_test_table_2 [id=59de2d64647446ab856b853639a3b1ca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:54.931116 5360 tablet_bootstrap.cc:492] T 371e10ffd8e64a33bbd950a140c7c514 P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:45:54.931245 5360 ts_tablet_manager.cc:1367] T 371e10ffd8e64a33bbd950a140c7c514 P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:54.931701 5360 raft_consensus.cc:348] T 371e10ffd8e64a33bbd950a140c7c514 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:54.931820 5360 raft_consensus.cc:374] T 371e10ffd8e64a33bbd950a140c7c514 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:54.931854 5360 raft_consensus.cc:729] T 371e10ffd8e64a33bbd950a140c7c514 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:45:54.931953 4750 tablet_service.cc:1467] Processing CreateTablet for tablet bdac13ec12c24d28abcca7f1e451451d (DEFAULT_TABLE table=rebalance_test_table_2 [id=59de2d64647446ab856b853639a3b1ca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:54.931985 5360 consensus_queue.cc:260] T 371e10ffd8e64a33bbd950a140c7c514 P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:54.932205 4750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdac13ec12c24d28abcca7f1e451451d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:54.932264 5360 ts_tablet_manager.cc:1398] T 371e10ffd8e64a33bbd950a140c7c514 P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:54.933297 5360 tablet_bootstrap.cc:492] T 85e3a77b9daf4f3f93a10b223b168064 P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:45:54.934113 5362 tablet_bootstrap.cc:492] T bdac13ec12c24d28abcca7f1e451451d P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:45:54.931125 4357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdac13ec12c24d28abcca7f1e451451d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:54.934909 5360 tablet_bootstrap.cc:654] T 85e3a77b9daf4f3f93a10b223b168064 P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:54.935015 5362 tablet_bootstrap.cc:654] T bdac13ec12c24d28abcca7f1e451451d P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:54.936028 5360 tablet_bootstrap.cc:492] T 85e3a77b9daf4f3f93a10b223b168064 P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:45:54.936125 5360 ts_tablet_manager.cc:1367] T 85e3a77b9daf4f3f93a10b223b168064 P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:45:54.936530 5360 raft_consensus.cc:348] T 85e3a77b9daf4f3f93a10b223b168064 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:54.936661 5360 raft_consensus.cc:374] T 85e3a77b9daf4f3f93a10b223b168064 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:54.936694 5360 raft_consensus.cc:729] T 85e3a77b9daf4f3f93a10b223b168064 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:45:54.936825 4749 tablet_service.cc:1467] Processing CreateTablet for tablet 85e3a77b9daf4f3f93a10b223b168064 (DEFAULT_TABLE table=rebalance_test_table_2 [id=59de2d64647446ab856b853639a3b1ca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:54.936839 5360 consensus_queue.cc:260] T 85e3a77b9daf4f3f93a10b223b168064 P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:54.937089 4749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85e3a77b9daf4f3f93a10b223b168064. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:54.937227 5360 ts_tablet_manager.cc:1398] T 85e3a77b9daf4f3f93a10b223b168064 P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:54.937393 5360 tablet_bootstrap.cc:492] T bdac13ec12c24d28abcca7f1e451451d P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:45:54.937587 5362 tablet_bootstrap.cc:492] T bdac13ec12c24d28abcca7f1e451451d P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:45:54.937656 5362 ts_tablet_manager.cc:1367] T bdac13ec12c24d28abcca7f1e451451d P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 20:45:54.937981 5360 tablet_bootstrap.cc:654] T bdac13ec12c24d28abcca7f1e451451d P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:54.938669 5362 raft_consensus.cc:348] T bdac13ec12c24d28abcca7f1e451451d P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:54.938776 5362 raft_consensus.cc:374] T bdac13ec12c24d28abcca7f1e451451d P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:54.938807 5362 raft_consensus.cc:729] T bdac13ec12c24d28abcca7f1e451451d P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:45:54.938926 5362 consensus_queue.cc:260] T bdac13ec12c24d28abcca7f1e451451d P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:54.939227 5362 ts_tablet_manager.cc:1398] T bdac13ec12c24d28abcca7f1e451451d P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:45:54.939376 5362 tablet_bootstrap.cc:492] T 85e3a77b9daf4f3f93a10b223b168064 P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:45:54.940057 5364 tablet_bootstrap.cc:492] T 371e10ffd8e64a33bbd950a140c7c514 P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:45:54.940146 5364 ts_tablet_manager.cc:1367] T 371e10ffd8e64a33bbd950a140c7c514 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.000s
I20240502 20:45:54.940440 5362 tablet_bootstrap.cc:654] T 85e3a77b9daf4f3f93a10b223b168064 P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:54.940672 5364 raft_consensus.cc:348] T 371e10ffd8e64a33bbd950a140c7c514 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:54.940822 5364 raft_consensus.cc:374] T 371e10ffd8e64a33bbd950a140c7c514 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:54.940857 5364 raft_consensus.cc:729] T 371e10ffd8e64a33bbd950a140c7c514 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:45:54.941025 5364 consensus_queue.cc:260] T 371e10ffd8e64a33bbd950a140c7c514 P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:54.941318 5364 ts_tablet_manager.cc:1398] T 371e10ffd8e64a33bbd950a140c7c514 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:54.941483 5364 tablet_bootstrap.cc:492] T bdac13ec12c24d28abcca7f1e451451d P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:45:54.941850 5362 tablet_bootstrap.cc:492] T 85e3a77b9daf4f3f93a10b223b168064 P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:45:54.941910 5362 ts_tablet_manager.cc:1367] T 85e3a77b9daf4f3f93a10b223b168064 P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:45:54.942212 5362 raft_consensus.cc:348] T 85e3a77b9daf4f3f93a10b223b168064 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:54.942276 5362 raft_consensus.cc:374] T 85e3a77b9daf4f3f93a10b223b168064 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:54.942296 5362 raft_consensus.cc:729] T 85e3a77b9daf4f3f93a10b223b168064 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:45:54.942411 5364 tablet_bootstrap.cc:654] T bdac13ec12c24d28abcca7f1e451451d P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:54.942394 5362 consensus_queue.cc:260] T 85e3a77b9daf4f3f93a10b223b168064 P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:54.942627 5362 ts_tablet_manager.cc:1398] T 85e3a77b9daf4f3f93a10b223b168064 P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:45:54.943732 5360 tablet_bootstrap.cc:492] T bdac13ec12c24d28abcca7f1e451451d P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:45:54.943796 5360 ts_tablet_manager.cc:1367] T bdac13ec12c24d28abcca7f1e451451d P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240502 20:45:54.944101 5360 raft_consensus.cc:348] T bdac13ec12c24d28abcca7f1e451451d P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:54.944223 5360 raft_consensus.cc:374] T bdac13ec12c24d28abcca7f1e451451d P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:54.944267 5360 raft_consensus.cc:729] T bdac13ec12c24d28abcca7f1e451451d P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:45:54.944380 5360 consensus_queue.cc:260] T bdac13ec12c24d28abcca7f1e451451d P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:54.944597 5360 ts_tablet_manager.cc:1398] T bdac13ec12c24d28abcca7f1e451451d P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:45:54.945135 5364 tablet_bootstrap.cc:492] T bdac13ec12c24d28abcca7f1e451451d P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:45:54.945212 5364 ts_tablet_manager.cc:1367] T bdac13ec12c24d28abcca7f1e451451d P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 20:45:54.945572 5364 raft_consensus.cc:348] T bdac13ec12c24d28abcca7f1e451451d P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:54.945672 5364 raft_consensus.cc:374] T bdac13ec12c24d28abcca7f1e451451d P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:54.945714 5364 raft_consensus.cc:729] T bdac13ec12c24d28abcca7f1e451451d P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:45:54.945823 5364 consensus_queue.cc:260] T bdac13ec12c24d28abcca7f1e451451d P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:54.946085 5364 ts_tablet_manager.cc:1398] T bdac13ec12c24d28abcca7f1e451451d P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:54.968807 5363 raft_consensus.cc:482] T 371e10ffd8e64a33bbd950a140c7c514 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:54.968943 5363 raft_consensus.cc:504] T 371e10ffd8e64a33bbd950a140c7c514 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:54.970018 5363 leader_election.cc:288] T 371e10ffd8e64a33bbd950a140c7c514 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097), ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:45:54.973601 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "371e10ffd8e64a33bbd950a140c7c514" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd" is_pre_election: true
I20240502 20:45:54.973654 4507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "371e10ffd8e64a33bbd950a140c7c514" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" is_pre_election: true
I20240502 20:45:54.973754 4770 raft_consensus.cc:2454] T 371e10ffd8e64a33bbd950a140c7c514 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 0.
I20240502 20:45:54.973839 4507 raft_consensus.cc:2454] T 371e10ffd8e64a33bbd950a140c7c514 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 0.
I20240502 20:45:54.974109 4313 leader_election.cc:302] T 371e10ffd8e64a33bbd950a140c7c514 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:45:54.974381 5363 raft_consensus.cc:2781] T 371e10ffd8e64a33bbd950a140c7c514 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:54.974440 5363 raft_consensus.cc:482] T 371e10ffd8e64a33bbd950a140c7c514 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:54.974467 5363 raft_consensus.cc:3037] T 371e10ffd8e64a33bbd950a140c7c514 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:54.975589 5363 raft_consensus.cc:504] T 371e10ffd8e64a33bbd950a140c7c514 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:54.975922 5363 leader_election.cc:288] T 371e10ffd8e64a33bbd950a140c7c514 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 election: Requested vote from peers c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097), ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:45:54.976222 4507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "371e10ffd8e64a33bbd950a140c7c514" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc"
I20240502 20:45:54.976238 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "371e10ffd8e64a33bbd950a140c7c514" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd"
I20240502 20:45:54.976330 4770 raft_consensus.cc:3037] T 371e10ffd8e64a33bbd950a140c7c514 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:54.976333 4507 raft_consensus.cc:3037] T 371e10ffd8e64a33bbd950a140c7c514 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:54.977301 4507 raft_consensus.cc:2454] T 371e10ffd8e64a33bbd950a140c7c514 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 1.
I20240502 20:45:54.977365 4770 raft_consensus.cc:2454] T 371e10ffd8e64a33bbd950a140c7c514 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 1.
I20240502 20:45:54.977613 4312 leader_election.cc:302] T 371e10ffd8e64a33bbd950a140c7c514 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3d26eafd51547acb780d4aac5ebe7cc, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:45:54.977795 5363 raft_consensus.cc:2781] T 371e10ffd8e64a33bbd950a140c7c514 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:54.977945 5363 raft_consensus.cc:686] T 371e10ffd8e64a33bbd950a140c7c514 P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEADER]: Becoming Leader. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Running, Role: LEADER
I20240502 20:45:54.978238 5363 consensus_queue.cc:237] T 371e10ffd8e64a33bbd950a140c7c514 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:54.980163 4235 catalog_manager.cc:5484] T 371e10ffd8e64a33bbd950a140c7c514 P c92e6e46782849479c8a0c6d82276fb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65). New cstate: current_term: 1 leader_uuid: "c92e6e46782849479c8a0c6d82276fb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:55.010758 5372 raft_consensus.cc:482] T bdac13ec12c24d28abcca7f1e451451d P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:55.010895 5372 raft_consensus.cc:504] T bdac13ec12c24d28abcca7f1e451451d P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:55.011874 5372 leader_election.cc:288] T bdac13ec12c24d28abcca7f1e451451d P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827), ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:45:55.015822 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdac13ec12c24d28abcca7f1e451451d" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd" is_pre_election: true
I20240502 20:45:55.015862 4377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdac13ec12c24d28abcca7f1e451451d" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2" is_pre_election: true
I20240502 20:45:55.015970 4770 raft_consensus.cc:2454] T bdac13ec12c24d28abcca7f1e451451d P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 0.
I20240502 20:45:55.016005 4377 raft_consensus.cc:2454] T bdac13ec12c24d28abcca7f1e451451d P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 0.
I20240502 20:45:55.016326 4444 leader_election.cc:302] T bdac13ec12c24d28abcca7f1e451451d P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3d26eafd51547acb780d4aac5ebe7cc, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:45:55.016587 5372 raft_consensus.cc:2781] T bdac13ec12c24d28abcca7f1e451451d P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:55.016647 5372 raft_consensus.cc:482] T bdac13ec12c24d28abcca7f1e451451d P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:55.016698 5372 raft_consensus.cc:3037] T bdac13ec12c24d28abcca7f1e451451d P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:55.017647 5372 raft_consensus.cc:504] T bdac13ec12c24d28abcca7f1e451451d P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:55.017985 5372 leader_election.cc:288] T bdac13ec12c24d28abcca7f1e451451d P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 election: Requested vote from peers c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827), ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:45:55.018375 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdac13ec12c24d28abcca7f1e451451d" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd"
I20240502 20:45:55.018370 4377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdac13ec12c24d28abcca7f1e451451d" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2"
I20240502 20:45:55.018483 4377 raft_consensus.cc:3037] T bdac13ec12c24d28abcca7f1e451451d P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:55.018486 4770 raft_consensus.cc:3037] T bdac13ec12c24d28abcca7f1e451451d P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:55.019279 4770 raft_consensus.cc:2454] T bdac13ec12c24d28abcca7f1e451451d P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 1.
I20240502 20:45:55.019466 4377 raft_consensus.cc:2454] T bdac13ec12c24d28abcca7f1e451451d P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 1.
I20240502 20:45:55.019791 4444 leader_election.cc:302] T bdac13ec12c24d28abcca7f1e451451d P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3d26eafd51547acb780d4aac5ebe7cc, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:45:55.020026 5372 raft_consensus.cc:2781] T bdac13ec12c24d28abcca7f1e451451d P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:55.020195 5372 raft_consensus.cc:686] T bdac13ec12c24d28abcca7f1e451451d P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Becoming Leader. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Running, Role: LEADER
I20240502 20:45:55.020485 5372 consensus_queue.cc:237] T bdac13ec12c24d28abcca7f1e451451d P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:55.022846 4235 catalog_manager.cc:5484] T bdac13ec12c24d28abcca7f1e451451d P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: term changed from 0 to 1, leader changed from <none> to c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66). New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } }
I20240502 20:45:55.072022 5363 raft_consensus.cc:482] T 85e3a77b9daf4f3f93a10b223b168064 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:55.072170 5363 raft_consensus.cc:504] T 85e3a77b9daf4f3f93a10b223b168064 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:55.072672 5363 leader_election.cc:288] T 85e3a77b9daf4f3f93a10b223b168064 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193), 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:45:55.073173 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e3a77b9daf4f3f93a10b223b168064" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd" is_pre_election: true
I20240502 20:45:55.073349 4770 raft_consensus.cc:2454] T 85e3a77b9daf4f3f93a10b223b168064 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 0.
I20240502 20:45:55.073706 4313 leader_election.cc:302] T 85e3a77b9daf4f3f93a10b223b168064 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:45:55.073879 5363 raft_consensus.cc:2781] T 85e3a77b9daf4f3f93a10b223b168064 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:55.073943 5363 raft_consensus.cc:482] T 85e3a77b9daf4f3f93a10b223b168064 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:55.073994 5363 raft_consensus.cc:3037] T 85e3a77b9daf4f3f93a10b223b168064 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:55.075066 5363 raft_consensus.cc:504] T 85e3a77b9daf4f3f93a10b223b168064 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:55.075414 5363 leader_election.cc:288] T 85e3a77b9daf4f3f93a10b223b168064 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 election: Requested vote from peers ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193), 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:45:55.075789 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e3a77b9daf4f3f93a10b223b168064" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd"
I20240502 20:45:55.075915 4770 raft_consensus.cc:3037] T 85e3a77b9daf4f3f93a10b223b168064 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:55.076651 4638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e3a77b9daf4f3f93a10b223b168064" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" is_pre_election: true
I20240502 20:45:55.076686 4639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e3a77b9daf4f3f93a10b223b168064" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91"
I20240502 20:45:55.076809 4638 raft_consensus.cc:2454] T 85e3a77b9daf4f3f93a10b223b168064 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 0.
I20240502 20:45:55.076859 4770 raft_consensus.cc:2454] T 85e3a77b9daf4f3f93a10b223b168064 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 1.
I20240502 20:45:55.077111 4313 leader_election.cc:302] T 85e3a77b9daf4f3f93a10b223b168064 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:45:55.077282 5363 raft_consensus.cc:2781] T 85e3a77b9daf4f3f93a10b223b168064 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:55.077353 5363 raft_consensus.cc:686] T 85e3a77b9daf4f3f93a10b223b168064 P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEADER]: Becoming Leader. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Running, Role: LEADER
I20240502 20:45:55.077504 5363 consensus_queue.cc:237] T 85e3a77b9daf4f3f93a10b223b168064 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:55.079150 4234 catalog_manager.cc:5484] T 85e3a77b9daf4f3f93a10b223b168064 P c92e6e46782849479c8a0c6d82276fb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65). New cstate: current_term: 1 leader_uuid: "c92e6e46782849479c8a0c6d82276fb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: HEALTHY } } }
W20240502 20:45:55.226186 5514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:55.226423 5514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:55.229621 5514 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 20:45:55.312697 4770 raft_consensus.cc:1261] T 85e3a77b9daf4f3f93a10b223b168064 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:55.313169 5363 consensus_queue.cc:1035] T 85e3a77b9daf4f3f93a10b223b168064 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:55.313658 4770 raft_consensus.cc:1261] T 371e10ffd8e64a33bbd950a140c7c514 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:55.314128 5363 consensus_queue.cc:1035] T 371e10ffd8e64a33bbd950a140c7c514 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:55.315928 4639 raft_consensus.cc:3037] T 85e3a77b9daf4f3f93a10b223b168064 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:55.317544 4377 raft_consensus.cc:1261] T bdac13ec12c24d28abcca7f1e451451d P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:55.317845 4770 raft_consensus.cc:1261] T bdac13ec12c24d28abcca7f1e451451d P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:55.318776 4507 raft_consensus.cc:1261] T 371e10ffd8e64a33bbd950a140c7c514 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:55.319180 5363 consensus_queue.cc:1035] T 371e10ffd8e64a33bbd950a140c7c514 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:55.319316 4639 raft_consensus.cc:1261] T 85e3a77b9daf4f3f93a10b223b168064 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:55.320328 5363 consensus_queue.cc:1035] T 85e3a77b9daf4f3f93a10b223b168064 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:55.321476 5372 consensus_queue.cc:1035] T bdac13ec12c24d28abcca7f1e451451d P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:55.321898 5372 consensus_queue.cc:1035] T bdac13ec12c24d28abcca7f1e451451d P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 113.597 ms
time per row: 0.0567986 ms
I20240502 20:45:55.389251 4231 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41812:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:45:55.395606 4619 tablet_service.cc:1467] Processing CreateTablet for tablet dbc1ef85e30d438682bc86935ceefd1b (DEFAULT_TABLE table=rebalance_test_table_3 [id=54d5208bcf1c43998e3ac81026ea9fbe]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:55.395783 4618 tablet_service.cc:1467] Processing CreateTablet for tablet 20373b53281a4214b27f014396f7a651 (DEFAULT_TABLE table=rebalance_test_table_3 [id=54d5208bcf1c43998e3ac81026ea9fbe]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:55.395956 4619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbc1ef85e30d438682bc86935ceefd1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:55.396184 4618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20373b53281a4214b27f014396f7a651. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:55.396324 4357 tablet_service.cc:1467] Processing CreateTablet for tablet dbc1ef85e30d438682bc86935ceefd1b (DEFAULT_TABLE table=rebalance_test_table_3 [id=54d5208bcf1c43998e3ac81026ea9fbe]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:55.396601 4357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbc1ef85e30d438682bc86935ceefd1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:55.398190 5366 tablet_bootstrap.cc:492] T 20373b53281a4214b27f014396f7a651 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:45:55.398806 5360 tablet_bootstrap.cc:492] T dbc1ef85e30d438682bc86935ceefd1b P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:45:55.399070 5366 tablet_bootstrap.cc:654] T 20373b53281a4214b27f014396f7a651 P 923cfab926e7443f8ffd7b21ad0a9d91: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:55.399678 5360 tablet_bootstrap.cc:654] T dbc1ef85e30d438682bc86935ceefd1b P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:55.400730 5360 tablet_bootstrap.cc:492] T dbc1ef85e30d438682bc86935ceefd1b P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:45:55.400807 5360 ts_tablet_manager.cc:1367] T dbc1ef85e30d438682bc86935ceefd1b P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:55.401154 4749 tablet_service.cc:1467] Processing CreateTablet for tablet 20373b53281a4214b27f014396f7a651 (DEFAULT_TABLE table=rebalance_test_table_3 [id=54d5208bcf1c43998e3ac81026ea9fbe]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:55.401239 5360 raft_consensus.cc:348] T dbc1ef85e30d438682bc86935ceefd1b P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:55.401340 5360 raft_consensus.cc:374] T dbc1ef85e30d438682bc86935ceefd1b P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:55.401371 5360 raft_consensus.cc:729] T dbc1ef85e30d438682bc86935ceefd1b P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:45:55.401427 4749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20373b53281a4214b27f014396f7a651. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:55.401482 5360 consensus_queue.cc:260] T dbc1ef85e30d438682bc86935ceefd1b P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:55.401531 4619 tablet_service.cc:1467] Processing CreateTablet for tablet 00eb91e624554baf918e96b098019be8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=54d5208bcf1c43998e3ac81026ea9fbe]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:55.401741 5360 ts_tablet_manager.cc:1398] T dbc1ef85e30d438682bc86935ceefd1b P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:55.401752 4619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00eb91e624554baf918e96b098019be8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:55.403537 5362 tablet_bootstrap.cc:492] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:45:55.403733 4750 tablet_service.cc:1467] Processing CreateTablet for tablet 00eb91e624554baf918e96b098019be8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=54d5208bcf1c43998e3ac81026ea9fbe]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:55.403966 4750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00eb91e624554baf918e96b098019be8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:55.404364 5362 tablet_bootstrap.cc:654] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:55.405695 4487 tablet_service.cc:1467] Processing CreateTablet for tablet 00eb91e624554baf918e96b098019be8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=54d5208bcf1c43998e3ac81026ea9fbe]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:55.405969 4487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00eb91e624554baf918e96b098019be8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:55.405978 4357 tablet_service.cc:1467] Processing CreateTablet for tablet 20373b53281a4214b27f014396f7a651 (DEFAULT_TABLE table=rebalance_test_table_3 [id=54d5208bcf1c43998e3ac81026ea9fbe]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:55.406219 4357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20373b53281a4214b27f014396f7a651. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:55.405706 4488 tablet_service.cc:1467] Processing CreateTablet for tablet dbc1ef85e30d438682bc86935ceefd1b (DEFAULT_TABLE table=rebalance_test_table_3 [id=54d5208bcf1c43998e3ac81026ea9fbe]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:55.407369 4488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbc1ef85e30d438682bc86935ceefd1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:55.407583 5362 tablet_bootstrap.cc:492] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:45:55.407666 5362 ts_tablet_manager.cc:1367] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 20:45:55.407904 5364 tablet_bootstrap.cc:492] T 00eb91e624554baf918e96b098019be8 P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:45:55.408084 5362 raft_consensus.cc:348] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:55.408186 5362 raft_consensus.cc:374] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:55.408233 5362 raft_consensus.cc:729] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:45:55.408372 5362 consensus_queue.cc:260] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:55.408675 5362 ts_tablet_manager.cc:1398] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:55.408684 5364 tablet_bootstrap.cc:654] T 00eb91e624554baf918e96b098019be8 P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:55.408831 5362 tablet_bootstrap.cc:492] T 00eb91e624554baf918e96b098019be8 P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:45:55.409711 5362 tablet_bootstrap.cc:654] T 00eb91e624554baf918e96b098019be8 P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:55.410718 5362 tablet_bootstrap.cc:492] T 00eb91e624554baf918e96b098019be8 P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:45:55.410847 5362 ts_tablet_manager.cc:1367] T 00eb91e624554baf918e96b098019be8 P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:55.411278 5362 raft_consensus.cc:348] T 00eb91e624554baf918e96b098019be8 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:55.411413 5362 raft_consensus.cc:374] T 00eb91e624554baf918e96b098019be8 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:55.411453 5362 raft_consensus.cc:729] T 00eb91e624554baf918e96b098019be8 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:45:55.411585 5362 consensus_queue.cc:260] T 00eb91e624554baf918e96b098019be8 P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:55.411834 5362 ts_tablet_manager.cc:1398] T 00eb91e624554baf918e96b098019be8 P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:55.414577 5360 tablet_bootstrap.cc:492] T 20373b53281a4214b27f014396f7a651 P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:45:55.415414 5364 tablet_bootstrap.cc:492] T 00eb91e624554baf918e96b098019be8 P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:45:55.415510 5364 ts_tablet_manager.cc:1367] T 00eb91e624554baf918e96b098019be8 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20240502 20:45:55.415644 5360 tablet_bootstrap.cc:654] T 20373b53281a4214b27f014396f7a651 P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:55.415932 5364 raft_consensus.cc:348] T 00eb91e624554baf918e96b098019be8 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:55.416028 5364 raft_consensus.cc:374] T 00eb91e624554baf918e96b098019be8 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:55.416062 5364 raft_consensus.cc:729] T 00eb91e624554baf918e96b098019be8 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:45:55.416189 5364 consensus_queue.cc:260] T 00eb91e624554baf918e96b098019be8 P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:55.416488 5364 ts_tablet_manager.cc:1398] T 00eb91e624554baf918e96b098019be8 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:45:55.416648 5364 tablet_bootstrap.cc:492] T dbc1ef85e30d438682bc86935ceefd1b P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:45:55.416793 5360 tablet_bootstrap.cc:492] T 20373b53281a4214b27f014396f7a651 P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:45:55.416858 5360 ts_tablet_manager.cc:1367] T 20373b53281a4214b27f014396f7a651 P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:55.417268 5360 raft_consensus.cc:348] T 20373b53281a4214b27f014396f7a651 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:55.417361 5360 raft_consensus.cc:374] T 20373b53281a4214b27f014396f7a651 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:55.417387 5360 raft_consensus.cc:729] T 20373b53281a4214b27f014396f7a651 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:45:55.417490 5360 consensus_queue.cc:260] T 20373b53281a4214b27f014396f7a651 P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:55.417618 5364 tablet_bootstrap.cc:654] T dbc1ef85e30d438682bc86935ceefd1b P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:55.417739 5360 ts_tablet_manager.cc:1398] T 20373b53281a4214b27f014396f7a651 P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:55.419479 5364 tablet_bootstrap.cc:492] T dbc1ef85e30d438682bc86935ceefd1b P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:45:55.419579 5364 ts_tablet_manager.cc:1367] T dbc1ef85e30d438682bc86935ceefd1b P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 20:45:55.420084 5364 raft_consensus.cc:348] T dbc1ef85e30d438682bc86935ceefd1b P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:55.420195 5364 raft_consensus.cc:374] T dbc1ef85e30d438682bc86935ceefd1b P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:55.420233 5364 raft_consensus.cc:729] T dbc1ef85e30d438682bc86935ceefd1b P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:45:55.420346 5364 consensus_queue.cc:260] T dbc1ef85e30d438682bc86935ceefd1b P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:55.420610 5364 ts_tablet_manager.cc:1398] T dbc1ef85e30d438682bc86935ceefd1b P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:45:55.421469 5366 tablet_bootstrap.cc:492] T 20373b53281a4214b27f014396f7a651 P 923cfab926e7443f8ffd7b21ad0a9d91: No bootstrap required, opened a new log
I20240502 20:45:55.421590 5366 ts_tablet_manager.cc:1367] T 20373b53281a4214b27f014396f7a651 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.001s
I20240502 20:45:55.422070 5366 raft_consensus.cc:348] T 20373b53281a4214b27f014396f7a651 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:55.422196 5366 raft_consensus.cc:374] T 20373b53281a4214b27f014396f7a651 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:55.422245 5366 raft_consensus.cc:729] T 20373b53281a4214b27f014396f7a651 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: FOLLOWER
I20240502 20:45:55.422370 5366 consensus_queue.cc:260] T 20373b53281a4214b27f014396f7a651 P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:55.422668 5366 ts_tablet_manager.cc:1398] T 20373b53281a4214b27f014396f7a651 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:45:55.422925 5366 tablet_bootstrap.cc:492] T dbc1ef85e30d438682bc86935ceefd1b P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:45:55.423805 5366 tablet_bootstrap.cc:654] T dbc1ef85e30d438682bc86935ceefd1b P 923cfab926e7443f8ffd7b21ad0a9d91: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:55.424801 5366 tablet_bootstrap.cc:492] T dbc1ef85e30d438682bc86935ceefd1b P 923cfab926e7443f8ffd7b21ad0a9d91: No bootstrap required, opened a new log
I20240502 20:45:55.424893 5366 ts_tablet_manager.cc:1367] T dbc1ef85e30d438682bc86935ceefd1b P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:45:55.425345 5366 raft_consensus.cc:348] T dbc1ef85e30d438682bc86935ceefd1b P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:55.425436 5366 raft_consensus.cc:374] T dbc1ef85e30d438682bc86935ceefd1b P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:55.425458 5366 raft_consensus.cc:729] T dbc1ef85e30d438682bc86935ceefd1b P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: FOLLOWER
I20240502 20:45:55.425565 5366 consensus_queue.cc:260] T dbc1ef85e30d438682bc86935ceefd1b P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:55.425799 5366 ts_tablet_manager.cc:1398] T dbc1ef85e30d438682bc86935ceefd1b P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:45:55.425977 5366 tablet_bootstrap.cc:492] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:45:55.426808 5366 tablet_bootstrap.cc:654] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:55.427914 5366 tablet_bootstrap.cc:492] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91: No bootstrap required, opened a new log
I20240502 20:45:55.427997 5366 ts_tablet_manager.cc:1367] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 20:45:55.428373 5366 raft_consensus.cc:348] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:55.428459 5366 raft_consensus.cc:374] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:55.428480 5366 raft_consensus.cc:729] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: FOLLOWER
I20240502 20:45:55.428558 5366 consensus_queue.cc:260] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:55.428802 5366 ts_tablet_manager.cc:1398] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:45:55.470132 5371 raft_consensus.cc:482] T dbc1ef85e30d438682bc86935ceefd1b P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:55.470254 5371 raft_consensus.cc:504] T dbc1ef85e30d438682bc86935ceefd1b P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:55.470584 5371 leader_election.cc:288] T dbc1ef85e30d438682bc86935ceefd1b P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827), c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097)
I20240502 20:45:55.470917 4377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbc1ef85e30d438682bc86935ceefd1b" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2" is_pre_election: true
I20240502 20:45:55.470988 4507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbc1ef85e30d438682bc86935ceefd1b" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" is_pre_election: true
I20240502 20:45:55.471066 4377 raft_consensus.cc:2454] T dbc1ef85e30d438682bc86935ceefd1b P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 0.
I20240502 20:45:55.471145 4507 raft_consensus.cc:2454] T dbc1ef85e30d438682bc86935ceefd1b P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 0.
I20240502 20:45:55.471374 4575 leader_election.cc:302] T dbc1ef85e30d438682bc86935ceefd1b P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:45:55.471571 5371 raft_consensus.cc:2781] T dbc1ef85e30d438682bc86935ceefd1b P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:55.471637 5371 raft_consensus.cc:482] T dbc1ef85e30d438682bc86935ceefd1b P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:55.471693 5371 raft_consensus.cc:3037] T dbc1ef85e30d438682bc86935ceefd1b P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:55.472482 5371 raft_consensus.cc:504] T dbc1ef85e30d438682bc86935ceefd1b P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:55.472826 5371 leader_election.cc:288] T dbc1ef85e30d438682bc86935ceefd1b P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 election: Requested vote from peers c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827), c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097)
I20240502 20:45:55.473140 4377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbc1ef85e30d438682bc86935ceefd1b" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2"
I20240502 20:45:55.473212 4507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbc1ef85e30d438682bc86935ceefd1b" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc"
I20240502 20:45:55.473269 4377 raft_consensus.cc:3037] T dbc1ef85e30d438682bc86935ceefd1b P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:55.473306 4507 raft_consensus.cc:3037] T dbc1ef85e30d438682bc86935ceefd1b P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:55.474020 4377 raft_consensus.cc:2454] T dbc1ef85e30d438682bc86935ceefd1b P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 1.
I20240502 20:45:55.474086 4507 raft_consensus.cc:2454] T dbc1ef85e30d438682bc86935ceefd1b P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 1.
I20240502 20:45:55.474269 4575 leader_election.cc:302] T dbc1ef85e30d438682bc86935ceefd1b P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:45:55.474455 5371 raft_consensus.cc:2781] T dbc1ef85e30d438682bc86935ceefd1b P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:55.474637 5371 raft_consensus.cc:686] T dbc1ef85e30d438682bc86935ceefd1b P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Becoming Leader. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Running, Role: LEADER
I20240502 20:45:55.474778 5371 consensus_queue.cc:237] T dbc1ef85e30d438682bc86935ceefd1b P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:55.476263 4233 catalog_manager.cc:5484] T dbc1ef85e30d438682bc86935ceefd1b P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67). New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:55.487498 5371 raft_consensus.cc:482] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:55.487583 5371 raft_consensus.cc:504] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:55.487900 5371 leader_election.cc:288] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097), ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:45:55.488214 4507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00eb91e624554baf918e96b098019be8" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" is_pre_election: true
I20240502 20:45:55.488343 4507 raft_consensus.cc:2454] T 00eb91e624554baf918e96b098019be8 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 0.
I20240502 20:45:55.488312 4769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00eb91e624554baf918e96b098019be8" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd" is_pre_election: true
I20240502 20:45:55.488440 4769 raft_consensus.cc:2454] T 00eb91e624554baf918e96b098019be8 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 0.
I20240502 20:45:55.488592 4574 leader_election.cc:302] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, c3d26eafd51547acb780d4aac5ebe7cc; no voters:
I20240502 20:45:55.488804 5371 raft_consensus.cc:2781] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:55.488850 5371 raft_consensus.cc:482] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:55.488876 5371 raft_consensus.cc:3037] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:55.489550 5371 raft_consensus.cc:504] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:55.489821 5371 leader_election.cc:288] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 election: Requested vote from peers c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097), ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:45:55.490139 4507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00eb91e624554baf918e96b098019be8" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc"
I20240502 20:45:55.490175 4769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00eb91e624554baf918e96b098019be8" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd"
I20240502 20:45:55.490264 4507 raft_consensus.cc:3037] T 00eb91e624554baf918e96b098019be8 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:55.490309 4769 raft_consensus.cc:3037] T 00eb91e624554baf918e96b098019be8 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:55.490967 4507 raft_consensus.cc:2454] T 00eb91e624554baf918e96b098019be8 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 1.
I20240502 20:45:55.491029 4769 raft_consensus.cc:2454] T 00eb91e624554baf918e96b098019be8 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 1.
I20240502 20:45:55.491218 4574 leader_election.cc:302] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, c3d26eafd51547acb780d4aac5ebe7cc; no voters:
I20240502 20:45:55.491379 5371 raft_consensus.cc:2781] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:55.491467 5371 raft_consensus.cc:686] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Becoming Leader. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Running, Role: LEADER
I20240502 20:45:55.491605 5371 consensus_queue.cc:237] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:55.493135 4233 catalog_manager.cc:5484] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67). New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:55.581377 5373 raft_consensus.cc:482] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:55.581503 5373 raft_consensus.cc:504] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:55.581892 5373 leader_election.cc:288] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141), c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:45:55.582346 4639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20373b53281a4214b27f014396f7a651" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" is_pre_election: true
I20240502 20:45:55.582381 4377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20373b53281a4214b27f014396f7a651" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2" is_pre_election: true
I20240502 20:45:55.582489 4639 raft_consensus.cc:2454] T 20373b53281a4214b27f014396f7a651 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 0.
I20240502 20:45:55.582502 4377 raft_consensus.cc:2454] T 20373b53281a4214b27f014396f7a651 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 0.
I20240502 20:45:55.582840 4703 leader_election.cc:302] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:45:55.583010 5373 raft_consensus.cc:2781] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:55.583072 5373 raft_consensus.cc:482] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:55.583110 5373 raft_consensus.cc:3037] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:55.583870 5373 raft_consensus.cc:504] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:55.584189 5373 leader_election.cc:288] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 election: Requested vote from peers 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141), c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:45:55.584501 4639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20373b53281a4214b27f014396f7a651" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91"
I20240502 20:45:55.584506 4377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20373b53281a4214b27f014396f7a651" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2"
I20240502 20:45:55.584615 4639 raft_consensus.cc:3037] T 20373b53281a4214b27f014396f7a651 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:55.584632 4377 raft_consensus.cc:3037] T 20373b53281a4214b27f014396f7a651 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:55.585417 4377 raft_consensus.cc:2454] T 20373b53281a4214b27f014396f7a651 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 1.
I20240502 20:45:55.585440 4639 raft_consensus.cc:2454] T 20373b53281a4214b27f014396f7a651 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 1.
I20240502 20:45:55.585692 4706 leader_election.cc:302] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:45:55.585848 5373 raft_consensus.cc:2781] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:55.585940 5373 raft_consensus.cc:686] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Becoming Leader. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Running, Role: LEADER
I20240502 20:45:55.586109 5373 consensus_queue.cc:237] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:55.588066 4233 catalog_manager.cc:5484] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd reported cstate change: term changed from 0 to 1, leader changed from <none> to ed47c023452a497192f35d2f617bf9bd (127.30.32.68). New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: UNKNOWN } } }
W20240502 20:45:55.703604 5564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:55.703851 5564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:55.706821 5564 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240502 20:45:55.800011 4507 raft_consensus.cc:1261] T dbc1ef85e30d438682bc86935ceefd1b P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:55.801069 4377 raft_consensus.cc:1261] T dbc1ef85e30d438682bc86935ceefd1b P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:55.801925 4639 raft_consensus.cc:1261] T 20373b53281a4214b27f014396f7a651 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:55.802035 5560 consensus_queue.cc:1035] T dbc1ef85e30d438682bc86935ceefd1b P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:55.802282 4377 raft_consensus.cc:1261] T 20373b53281a4214b27f014396f7a651 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:55.802381 4769 raft_consensus.cc:1261] T 00eb91e624554baf918e96b098019be8 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:55.802721 5373 consensus_queue.cc:1035] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:55.802783 5560 consensus_queue.cc:1035] T dbc1ef85e30d438682bc86935ceefd1b P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:55.803074 5560 consensus_queue.cc:1035] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:55.803279 5373 consensus_queue.cc:1035] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:55.805348 4507 raft_consensus.cc:1261] T 00eb91e624554baf918e96b098019be8 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:55.806953 5371 consensus_queue.cc:1035] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 118.158 ms
time per row: 0.0590792 ms
I20240502 20:45:55.868907 4227 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41812:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:45:55.874622 4488 tablet_service.cc:1467] Processing CreateTablet for tablet 0982b98bfed242eba152de225a0158a2 (DEFAULT_TABLE table=rebalance_test_table_4 [id=129781998e46479d8beddda94bcb5a8f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:55.874683 4619 tablet_service.cc:1467] Processing CreateTablet for tablet 0982b98bfed242eba152de225a0158a2 (DEFAULT_TABLE table=rebalance_test_table_4 [id=129781998e46479d8beddda94bcb5a8f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:55.874962 4488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0982b98bfed242eba152de225a0158a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:55.874960 4619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0982b98bfed242eba152de225a0158a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:55.875661 4357 tablet_service.cc:1467] Processing CreateTablet for tablet 0982b98bfed242eba152de225a0158a2 (DEFAULT_TABLE table=rebalance_test_table_4 [id=129781998e46479d8beddda94bcb5a8f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:55.875905 4357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0982b98bfed242eba152de225a0158a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:55.877096 5366 tablet_bootstrap.cc:492] T 0982b98bfed242eba152de225a0158a2 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:45:55.877112 5364 tablet_bootstrap.cc:492] T 0982b98bfed242eba152de225a0158a2 P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:45:55.877894 5366 tablet_bootstrap.cc:654] T 0982b98bfed242eba152de225a0158a2 P 923cfab926e7443f8ffd7b21ad0a9d91: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:55.877894 5364 tablet_bootstrap.cc:654] T 0982b98bfed242eba152de225a0158a2 P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:55.878002 4355 tablet_service.cc:1467] Processing CreateTablet for tablet fa8eb41a63c04e3589ac94ae13a87dc9 (DEFAULT_TABLE table=rebalance_test_table_4 [id=129781998e46479d8beddda94bcb5a8f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:55.878202 4355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa8eb41a63c04e3589ac94ae13a87dc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:55.878439 5360 tablet_bootstrap.cc:492] T 0982b98bfed242eba152de225a0158a2 P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:45:55.879014 4619 tablet_service.cc:1467] Processing CreateTablet for tablet f1f81664a2ee49ef8c775608d3a3d2c2 (DEFAULT_TABLE table=rebalance_test_table_4 [id=129781998e46479d8beddda94bcb5a8f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:55.879241 4619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1f81664a2ee49ef8c775608d3a3d2c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:55.879243 5360 tablet_bootstrap.cc:654] T 0982b98bfed242eba152de225a0158a2 P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:55.879979 4356 tablet_service.cc:1467] Processing CreateTablet for tablet f1f81664a2ee49ef8c775608d3a3d2c2 (DEFAULT_TABLE table=rebalance_test_table_4 [id=129781998e46479d8beddda94bcb5a8f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:55.880164 4356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1f81664a2ee49ef8c775608d3a3d2c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:55.880244 4750 tablet_service.cc:1467] Processing CreateTablet for tablet fa8eb41a63c04e3589ac94ae13a87dc9 (DEFAULT_TABLE table=rebalance_test_table_4 [id=129781998e46479d8beddda94bcb5a8f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:55.880497 4750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa8eb41a63c04e3589ac94ae13a87dc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:55.881765 5360 tablet_bootstrap.cc:492] T 0982b98bfed242eba152de225a0158a2 P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:45:55.881839 5360 ts_tablet_manager.cc:1367] T 0982b98bfed242eba152de225a0158a2 P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 20:45:55.882400 5360 raft_consensus.cc:348] T 0982b98bfed242eba152de225a0158a2 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:55.882498 5360 raft_consensus.cc:374] T 0982b98bfed242eba152de225a0158a2 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:55.882529 5360 raft_consensus.cc:729] T 0982b98bfed242eba152de225a0158a2 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:45:55.882647 5360 consensus_queue.cc:260] T 0982b98bfed242eba152de225a0158a2 P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:55.882855 4749 tablet_service.cc:1467] Processing CreateTablet for tablet f1f81664a2ee49ef8c775608d3a3d2c2 (DEFAULT_TABLE table=rebalance_test_table_4 [id=129781998e46479d8beddda94bcb5a8f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:55.882928 5360 ts_tablet_manager.cc:1398] T 0982b98bfed242eba152de225a0158a2 P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:55.883085 5360 tablet_bootstrap.cc:492] T fa8eb41a63c04e3589ac94ae13a87dc9 P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:45:55.883085 4749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1f81664a2ee49ef8c775608d3a3d2c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:55.884059 5360 tablet_bootstrap.cc:654] T fa8eb41a63c04e3589ac94ae13a87dc9 P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:55.885186 5362 tablet_bootstrap.cc:492] T fa8eb41a63c04e3589ac94ae13a87dc9 P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:45:55.885195 5366 tablet_bootstrap.cc:492] T 0982b98bfed242eba152de225a0158a2 P 923cfab926e7443f8ffd7b21ad0a9d91: No bootstrap required, opened a new log
I20240502 20:45:55.885267 5366 ts_tablet_manager.cc:1367] T 0982b98bfed242eba152de225a0158a2 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240502 20:45:55.885720 5366 raft_consensus.cc:348] T 0982b98bfed242eba152de225a0158a2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:55.885818 5366 raft_consensus.cc:374] T 0982b98bfed242eba152de225a0158a2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:55.885850 5366 raft_consensus.cc:729] T 0982b98bfed242eba152de225a0158a2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: FOLLOWER
I20240502 20:45:55.885973 5366 consensus_queue.cc:260] T 0982b98bfed242eba152de225a0158a2 P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:55.886059 5362 tablet_bootstrap.cc:654] T fa8eb41a63c04e3589ac94ae13a87dc9 P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:55.886305 5366 ts_tablet_manager.cc:1398] T 0982b98bfed242eba152de225a0158a2 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:55.886476 5366 tablet_bootstrap.cc:492] T f1f81664a2ee49ef8c775608d3a3d2c2 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:45:55.887356 5366 tablet_bootstrap.cc:654] T f1f81664a2ee49ef8c775608d3a3d2c2 P 923cfab926e7443f8ffd7b21ad0a9d91: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:55.888048 4487 tablet_service.cc:1467] Processing CreateTablet for tablet fa8eb41a63c04e3589ac94ae13a87dc9 (DEFAULT_TABLE table=rebalance_test_table_4 [id=129781998e46479d8beddda94bcb5a8f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:55.888309 4487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa8eb41a63c04e3589ac94ae13a87dc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:55.890821 5362 tablet_bootstrap.cc:492] T fa8eb41a63c04e3589ac94ae13a87dc9 P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:45:55.890910 5362 ts_tablet_manager.cc:1367] T fa8eb41a63c04e3589ac94ae13a87dc9 P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 20:45:55.891106 5360 tablet_bootstrap.cc:492] T fa8eb41a63c04e3589ac94ae13a87dc9 P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:45:55.891185 5360 ts_tablet_manager.cc:1367] T fa8eb41a63c04e3589ac94ae13a87dc9 P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 20:45:55.891284 5362 raft_consensus.cc:348] T fa8eb41a63c04e3589ac94ae13a87dc9 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:55.891386 5362 raft_consensus.cc:374] T fa8eb41a63c04e3589ac94ae13a87dc9 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:55.891412 5362 raft_consensus.cc:729] T fa8eb41a63c04e3589ac94ae13a87dc9 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:45:55.891516 5362 consensus_queue.cc:260] T fa8eb41a63c04e3589ac94ae13a87dc9 P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:55.891602 5360 raft_consensus.cc:348] T fa8eb41a63c04e3589ac94ae13a87dc9 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:55.891712 5360 raft_consensus.cc:374] T fa8eb41a63c04e3589ac94ae13a87dc9 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:55.891747 5360 raft_consensus.cc:729] T fa8eb41a63c04e3589ac94ae13a87dc9 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:45:55.891804 5362 ts_tablet_manager.cc:1398] T fa8eb41a63c04e3589ac94ae13a87dc9 P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:55.891865 5360 consensus_queue.cc:260] T fa8eb41a63c04e3589ac94ae13a87dc9 P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:55.891952 5362 tablet_bootstrap.cc:492] T f1f81664a2ee49ef8c775608d3a3d2c2 P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:45:55.892757 5366 tablet_bootstrap.cc:492] T f1f81664a2ee49ef8c775608d3a3d2c2 P 923cfab926e7443f8ffd7b21ad0a9d91: No bootstrap required, opened a new log
I20240502 20:45:55.892846 5366 ts_tablet_manager.cc:1367] T f1f81664a2ee49ef8c775608d3a3d2c2 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240502 20:45:55.892134 5360 ts_tablet_manager.cc:1398] T fa8eb41a63c04e3589ac94ae13a87dc9 P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:45:55.893244 5360 tablet_bootstrap.cc:492] T f1f81664a2ee49ef8c775608d3a3d2c2 P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:45:55.893301 5362 tablet_bootstrap.cc:654] T f1f81664a2ee49ef8c775608d3a3d2c2 P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:55.893373 5366 raft_consensus.cc:348] T f1f81664a2ee49ef8c775608d3a3d2c2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:55.893478 5366 raft_consensus.cc:374] T f1f81664a2ee49ef8c775608d3a3d2c2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:55.893515 5366 raft_consensus.cc:729] T f1f81664a2ee49ef8c775608d3a3d2c2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: FOLLOWER
I20240502 20:45:55.893651 5366 consensus_queue.cc:260] T f1f81664a2ee49ef8c775608d3a3d2c2 P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:55.893932 5366 ts_tablet_manager.cc:1398] T f1f81664a2ee49ef8c775608d3a3d2c2 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:55.894063 5603 raft_consensus.cc:482] T 0982b98bfed242eba152de225a0158a2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:55.894173 5603 raft_consensus.cc:504] T 0982b98bfed242eba152de225a0158a2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:55.894440 5364 tablet_bootstrap.cc:492] T 0982b98bfed242eba152de225a0158a2 P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:45:55.894544 5364 ts_tablet_manager.cc:1367] T 0982b98bfed242eba152de225a0158a2 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.001s
I20240502 20:45:55.894587 5603 leader_election.cc:288] T 0982b98bfed242eba152de225a0158a2 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097), c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:45:55.894901 4375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0982b98bfed242eba152de225a0158a2" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2" is_pre_election: true
I20240502 20:45:55.895028 4375 raft_consensus.cc:2454] T 0982b98bfed242eba152de225a0158a2 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 0.
I20240502 20:45:55.895062 5364 raft_consensus.cc:348] T 0982b98bfed242eba152de225a0158a2 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:55.895180 5364 raft_consensus.cc:374] T 0982b98bfed242eba152de225a0158a2 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:55.895223 5364 raft_consensus.cc:729] T 0982b98bfed242eba152de225a0158a2 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:45:55.895308 4575 leader_election.cc:302] T 0982b98bfed242eba152de225a0158a2 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:45:55.895344 5364 consensus_queue.cc:260] T 0982b98bfed242eba152de225a0158a2 P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:55.895543 5360 tablet_bootstrap.cc:654] T f1f81664a2ee49ef8c775608d3a3d2c2 P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:55.895591 5603 raft_consensus.cc:2781] T 0982b98bfed242eba152de225a0158a2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:55.895627 5364 ts_tablet_manager.cc:1398] T 0982b98bfed242eba152de225a0158a2 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:55.895648 5603 raft_consensus.cc:482] T 0982b98bfed242eba152de225a0158a2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:55.895695 5603 raft_consensus.cc:3037] T 0982b98bfed242eba152de225a0158a2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:55.895823 5364 tablet_bootstrap.cc:492] T fa8eb41a63c04e3589ac94ae13a87dc9 P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:45:55.896723 5603 raft_consensus.cc:504] T 0982b98bfed242eba152de225a0158a2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:55.896878 4507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0982b98bfed242eba152de225a0158a2" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" is_pre_election: true
I20240502 20:45:55.896937 5364 tablet_bootstrap.cc:654] T fa8eb41a63c04e3589ac94ae13a87dc9 P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:55.897006 4507 raft_consensus.cc:2454] T 0982b98bfed242eba152de225a0158a2 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 0.
I20240502 20:45:55.897504 4507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0982b98bfed242eba152de225a0158a2" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc"
I20240502 20:45:55.897609 4507 raft_consensus.cc:3037] T 0982b98bfed242eba152de225a0158a2 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:55.897661 4375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0982b98bfed242eba152de225a0158a2" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2"
I20240502 20:45:55.897753 4375 raft_consensus.cc:3037] T 0982b98bfed242eba152de225a0158a2 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:55.898012 5364 tablet_bootstrap.cc:492] T fa8eb41a63c04e3589ac94ae13a87dc9 P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:45:55.898082 5364 ts_tablet_manager.cc:1367] T fa8eb41a63c04e3589ac94ae13a87dc9 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:45:55.898505 5364 raft_consensus.cc:348] T fa8eb41a63c04e3589ac94ae13a87dc9 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:55.898609 5364 raft_consensus.cc:374] T fa8eb41a63c04e3589ac94ae13a87dc9 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:55.898643 5364 raft_consensus.cc:729] T fa8eb41a63c04e3589ac94ae13a87dc9 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:45:55.898664 4375 raft_consensus.cc:2454] T 0982b98bfed242eba152de225a0158a2 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 1.
I20240502 20:45:55.898767 5364 consensus_queue.cc:260] T fa8eb41a63c04e3589ac94ae13a87dc9 P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:55.898969 4575 leader_election.cc:302] T 0982b98bfed242eba152de225a0158a2 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:45:55.899063 5364 ts_tablet_manager.cc:1398] T fa8eb41a63c04e3589ac94ae13a87dc9 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:55.899147 5560 raft_consensus.cc:2781] T 0982b98bfed242eba152de225a0158a2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:55.899273 5560 raft_consensus.cc:686] T 0982b98bfed242eba152de225a0158a2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Becoming Leader. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Running, Role: LEADER
I20240502 20:45:55.899423 5560 consensus_queue.cc:237] T 0982b98bfed242eba152de225a0158a2 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:55.899609 5362 tablet_bootstrap.cc:492] T f1f81664a2ee49ef8c775608d3a3d2c2 P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:45:55.899708 5362 ts_tablet_manager.cc:1367] T f1f81664a2ee49ef8c775608d3a3d2c2 P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 20:45:55.899750 5360 tablet_bootstrap.cc:492] T f1f81664a2ee49ef8c775608d3a3d2c2 P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:45:55.899827 5360 ts_tablet_manager.cc:1367] T f1f81664a2ee49ef8c775608d3a3d2c2 P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 20:45:55.900133 5362 raft_consensus.cc:348] T f1f81664a2ee49ef8c775608d3a3d2c2 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:55.900236 5362 raft_consensus.cc:374] T f1f81664a2ee49ef8c775608d3a3d2c2 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:55.900278 5362 raft_consensus.cc:729] T f1f81664a2ee49ef8c775608d3a3d2c2 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:45:55.900246 5360 raft_consensus.cc:348] T f1f81664a2ee49ef8c775608d3a3d2c2 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:55.900336 5360 raft_consensus.cc:374] T f1f81664a2ee49ef8c775608d3a3d2c2 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:55.900368 5360 raft_consensus.cc:729] T f1f81664a2ee49ef8c775608d3a3d2c2 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:45:55.900391 5362 consensus_queue.cc:260] T f1f81664a2ee49ef8c775608d3a3d2c2 P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:55.900492 5360 consensus_queue.cc:260] T f1f81664a2ee49ef8c775608d3a3d2c2 P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:55.900628 5380 raft_consensus.cc:482] T f1f81664a2ee49ef8c775608d3a3d2c2 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:55.900691 5362 ts_tablet_manager.cc:1398] T f1f81664a2ee49ef8c775608d3a3d2c2 P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:45:55.900784 5360 ts_tablet_manager.cc:1398] T f1f81664a2ee49ef8c775608d3a3d2c2 P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:55.900696 5380 raft_consensus.cc:504] T f1f81664a2ee49ef8c775608d3a3d2c2 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:55.898510 4507 raft_consensus.cc:2454] T 0982b98bfed242eba152de225a0158a2 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 1.
I20240502 20:45:55.901185 5380 leader_election.cc:288] T f1f81664a2ee49ef8c775608d3a3d2c2 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141), c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:45:55.901619 5603 leader_election.cc:288] T 0982b98bfed242eba152de225a0158a2 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 election: Requested vote from peers c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097), c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:45:55.901886 4375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1f81664a2ee49ef8c775608d3a3d2c2" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2" is_pre_election: true
I20240502 20:45:55.901932 4639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1f81664a2ee49ef8c775608d3a3d2c2" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" is_pre_election: true
I20240502 20:45:55.901996 4375 raft_consensus.cc:2454] T f1f81664a2ee49ef8c775608d3a3d2c2 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 0.
I20240502 20:45:55.902036 4639 raft_consensus.cc:2454] T f1f81664a2ee49ef8c775608d3a3d2c2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 0.
I20240502 20:45:55.902302 4706 leader_election.cc:302] T f1f81664a2ee49ef8c775608d3a3d2c2 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:45:55.903733 5606 raft_consensus.cc:2781] T f1f81664a2ee49ef8c775608d3a3d2c2 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:55.903805 5606 raft_consensus.cc:482] T f1f81664a2ee49ef8c775608d3a3d2c2 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:55.903870 5606 raft_consensus.cc:3037] T f1f81664a2ee49ef8c775608d3a3d2c2 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:55.904606 4230 catalog_manager.cc:5484] T 0982b98bfed242eba152de225a0158a2 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67). New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:55.904820 5606 raft_consensus.cc:504] T f1f81664a2ee49ef8c775608d3a3d2c2 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:55.905181 5606 leader_election.cc:288] T f1f81664a2ee49ef8c775608d3a3d2c2 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 election: Requested vote from peers 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141), c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:45:55.905484 4375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1f81664a2ee49ef8c775608d3a3d2c2" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2"
I20240502 20:45:55.905484 4639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1f81664a2ee49ef8c775608d3a3d2c2" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91"
I20240502 20:45:55.905586 4375 raft_consensus.cc:3037] T f1f81664a2ee49ef8c775608d3a3d2c2 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:55.905586 4639 raft_consensus.cc:3037] T f1f81664a2ee49ef8c775608d3a3d2c2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:55.906437 4375 raft_consensus.cc:2454] T f1f81664a2ee49ef8c775608d3a3d2c2 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 1.
I20240502 20:45:55.906437 4639 raft_consensus.cc:2454] T f1f81664a2ee49ef8c775608d3a3d2c2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 1.
I20240502 20:45:55.906744 4703 leader_election.cc:302] T f1f81664a2ee49ef8c775608d3a3d2c2 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:45:55.907199 5606 raft_consensus.cc:2781] T f1f81664a2ee49ef8c775608d3a3d2c2 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:55.907295 5606 raft_consensus.cc:686] T f1f81664a2ee49ef8c775608d3a3d2c2 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Becoming Leader. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Running, Role: LEADER
I20240502 20:45:55.907415 5606 consensus_queue.cc:237] T f1f81664a2ee49ef8c775608d3a3d2c2 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:55.908782 4228 catalog_manager.cc:5484] T f1f81664a2ee49ef8c775608d3a3d2c2 P ed47c023452a497192f35d2f617bf9bd reported cstate change: term changed from 0 to 1, leader changed from <none> to ed47c023452a497192f35d2f617bf9bd (127.30.32.68). New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:55.954960 5606 raft_consensus.cc:482] T fa8eb41a63c04e3589ac94ae13a87dc9 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:55.955106 5606 raft_consensus.cc:504] T fa8eb41a63c04e3589ac94ae13a87dc9 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:55.955528 5606 leader_election.cc:288] T fa8eb41a63c04e3589ac94ae13a87dc9 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827), c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097)
I20240502 20:45:55.955989 4375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa8eb41a63c04e3589ac94ae13a87dc9" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2" is_pre_election: true
I20240502 20:45:55.956001 4507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa8eb41a63c04e3589ac94ae13a87dc9" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" is_pre_election: true
I20240502 20:45:55.956127 4507 raft_consensus.cc:2454] T fa8eb41a63c04e3589ac94ae13a87dc9 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 0.
I20240502 20:45:55.956143 4375 raft_consensus.cc:2454] T fa8eb41a63c04e3589ac94ae13a87dc9 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 0.
I20240502 20:45:55.956449 4706 leader_election.cc:302] T fa8eb41a63c04e3589ac94ae13a87dc9 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:45:55.956668 5606 raft_consensus.cc:2781] T fa8eb41a63c04e3589ac94ae13a87dc9 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:55.956743 5606 raft_consensus.cc:482] T fa8eb41a63c04e3589ac94ae13a87dc9 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:55.956789 5606 raft_consensus.cc:3037] T fa8eb41a63c04e3589ac94ae13a87dc9 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:55.957935 5606 raft_consensus.cc:504] T fa8eb41a63c04e3589ac94ae13a87dc9 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:55.958242 5606 leader_election.cc:288] T fa8eb41a63c04e3589ac94ae13a87dc9 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 election: Requested vote from peers c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827), c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097)
I20240502 20:45:55.958564 4375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa8eb41a63c04e3589ac94ae13a87dc9" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2"
I20240502 20:45:55.958628 4507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa8eb41a63c04e3589ac94ae13a87dc9" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc"
I20240502 20:45:55.958703 4375 raft_consensus.cc:3037] T fa8eb41a63c04e3589ac94ae13a87dc9 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:55.958747 4507 raft_consensus.cc:3037] T fa8eb41a63c04e3589ac94ae13a87dc9 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:55.959440 4507 raft_consensus.cc:2454] T fa8eb41a63c04e3589ac94ae13a87dc9 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 1.
I20240502 20:45:55.959444 4375 raft_consensus.cc:2454] T fa8eb41a63c04e3589ac94ae13a87dc9 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 1.
I20240502 20:45:55.959697 4706 leader_election.cc:302] T fa8eb41a63c04e3589ac94ae13a87dc9 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:45:55.959901 5606 raft_consensus.cc:2781] T fa8eb41a63c04e3589ac94ae13a87dc9 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:55.959983 5606 raft_consensus.cc:686] T fa8eb41a63c04e3589ac94ae13a87dc9 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Becoming Leader. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Running, Role: LEADER
I20240502 20:45:55.960121 5606 consensus_queue.cc:237] T fa8eb41a63c04e3589ac94ae13a87dc9 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:55.961803 4228 catalog_manager.cc:5484] T fa8eb41a63c04e3589ac94ae13a87dc9 P ed47c023452a497192f35d2f617bf9bd reported cstate change: term changed from 0 to 1, leader changed from <none> to ed47c023452a497192f35d2f617bf9bd (127.30.32.68). New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: UNKNOWN } } }
W20240502 20:45:56.106038 5621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:56.106276 5621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:56.109256 5621 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240502 20:45:56.198201 4375 raft_consensus.cc:1261] T 0982b98bfed242eba152de225a0158a2 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:56.198789 5560 consensus_queue.cc:1035] T 0982b98bfed242eba152de225a0158a2 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:56.199282 4507 raft_consensus.cc:1261] T 0982b98bfed242eba152de225a0158a2 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:56.202123 5603 consensus_queue.cc:1035] T 0982b98bfed242eba152de225a0158a2 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:56.204017 4377 raft_consensus.cc:1261] T f1f81664a2ee49ef8c775608d3a3d2c2 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:56.204631 4639 raft_consensus.cc:1261] T f1f81664a2ee49ef8c775608d3a3d2c2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:56.204958 5606 consensus_queue.cc:1035] T f1f81664a2ee49ef8c775608d3a3d2c2 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:56.205564 5606 consensus_queue.cc:1035] T f1f81664a2ee49ef8c775608d3a3d2c2 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:56.206393 4506 raft_consensus.cc:1261] T fa8eb41a63c04e3589ac94ae13a87dc9 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:56.206751 5373 consensus_queue.cc:1035] T fa8eb41a63c04e3589ac94ae13a87dc9 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:56.207762 4376 raft_consensus.cc:1261] T fa8eb41a63c04e3589ac94ae13a87dc9 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:56.213377 5380 consensus_queue.cc:1035] T fa8eb41a63c04e3589ac94ae13a87dc9 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:56.230616 5488 mvcc.cc:205] Tried to move back new op lower bound from 7023340569396867072 to 7023340568197877760. Current Snapshot: MvccSnapshot[applied={T|T < 7023340569396867072}]
INSERT report
rows total: 2000
time total: 116.856 ms
time per row: 0.0584282 ms
I20240502 20:45:56.261885 5431 mvcc.cc:205] Tried to move back new op lower bound from 7023340569430495232 to 7023340568414031872. Current Snapshot: MvccSnapshot[applied={T|T < 7023340569399242752}]
I20240502 20:45:56.273208 4233 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41812:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:45:56.278440 4619 tablet_service.cc:1467] Processing CreateTablet for tablet d37fab152a8a44868d3160ec3057bb98 (DEFAULT_TABLE table=rebalance_test_table_5 [id=5c60c96389ff4cf3bbd5edb6a694c70d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:56.278705 4618 tablet_service.cc:1467] Processing CreateTablet for tablet 5053a7afb4994ecdbc92b129c58903cf (DEFAULT_TABLE table=rebalance_test_table_5 [id=5c60c96389ff4cf3bbd5edb6a694c70d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:56.278774 4619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d37fab152a8a44868d3160ec3057bb98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:56.278944 4618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5053a7afb4994ecdbc92b129c58903cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:56.279294 4749 tablet_service.cc:1467] Processing CreateTablet for tablet d37fab152a8a44868d3160ec3057bb98 (DEFAULT_TABLE table=rebalance_test_table_5 [id=5c60c96389ff4cf3bbd5edb6a694c70d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:56.279335 4617 tablet_service.cc:1467] Processing CreateTablet for tablet 253d158b340a444182793f18239bfa22 (DEFAULT_TABLE table=rebalance_test_table_5 [id=5c60c96389ff4cf3bbd5edb6a694c70d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:56.279433 4487 tablet_service.cc:1467] Processing CreateTablet for tablet d37fab152a8a44868d3160ec3057bb98 (DEFAULT_TABLE table=rebalance_test_table_5 [id=5c60c96389ff4cf3bbd5edb6a694c70d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:56.279544 4617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 253d158b340a444182793f18239bfa22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:56.279557 4749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d37fab152a8a44868d3160ec3057bb98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:56.279681 4487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d37fab152a8a44868d3160ec3057bb98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:56.281706 5362 tablet_bootstrap.cc:492] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:45:56.281771 5366 tablet_bootstrap.cc:492] T 5053a7afb4994ecdbc92b129c58903cf P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:45:56.281898 4488 tablet_service.cc:1467] Processing CreateTablet for tablet 5053a7afb4994ecdbc92b129c58903cf (DEFAULT_TABLE table=rebalance_test_table_5 [id=5c60c96389ff4cf3bbd5edb6a694c70d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:56.282145 4488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5053a7afb4994ecdbc92b129c58903cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:56.282465 5362 tablet_bootstrap.cc:654] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:56.282652 5366 tablet_bootstrap.cc:654] T 5053a7afb4994ecdbc92b129c58903cf P 923cfab926e7443f8ffd7b21ad0a9d91: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:56.283001 5364 tablet_bootstrap.cc:492] T d37fab152a8a44868d3160ec3057bb98 P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:45:56.283979 5364 tablet_bootstrap.cc:654] T d37fab152a8a44868d3160ec3057bb98 P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:56.284382 4486 tablet_service.cc:1467] Processing CreateTablet for tablet 253d158b340a444182793f18239bfa22 (DEFAULT_TABLE table=rebalance_test_table_5 [id=5c60c96389ff4cf3bbd5edb6a694c70d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:56.284615 4486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 253d158b340a444182793f18239bfa22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:56.284829 5362 tablet_bootstrap.cc:492] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:45:56.284893 5362 ts_tablet_manager.cc:1367] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 20:45:56.285352 5362 raft_consensus.cc:348] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:56.285460 5362 raft_consensus.cc:374] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:56.285496 5362 raft_consensus.cc:729] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:45:56.285626 5362 consensus_queue.cc:260] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:56.285920 5362 ts_tablet_manager.cc:1398] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:56.286767 5364 tablet_bootstrap.cc:492] T d37fab152a8a44868d3160ec3057bb98 P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:45:56.286845 5364 ts_tablet_manager.cc:1367] T d37fab152a8a44868d3160ec3057bb98 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 20:45:56.286782 4749 tablet_service.cc:1467] Processing CreateTablet for tablet 5053a7afb4994ecdbc92b129c58903cf (DEFAULT_TABLE table=rebalance_test_table_5 [id=5c60c96389ff4cf3bbd5edb6a694c70d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:56.287093 4749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5053a7afb4994ecdbc92b129c58903cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:56.287035 4356 tablet_service.cc:1467] Processing CreateTablet for tablet 253d158b340a444182793f18239bfa22 (DEFAULT_TABLE table=rebalance_test_table_5 [id=5c60c96389ff4cf3bbd5edb6a694c70d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:56.287197 5364 raft_consensus.cc:348] T d37fab152a8a44868d3160ec3057bb98 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:56.287277 5364 raft_consensus.cc:374] T d37fab152a8a44868d3160ec3057bb98 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:56.287298 5364 raft_consensus.cc:729] T d37fab152a8a44868d3160ec3057bb98 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:45:56.287303 4356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 253d158b340a444182793f18239bfa22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:56.287371 5364 consensus_queue.cc:260] T d37fab152a8a44868d3160ec3057bb98 P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:56.287602 5364 ts_tablet_manager.cc:1398] T d37fab152a8a44868d3160ec3057bb98 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:56.287714 5364 tablet_bootstrap.cc:492] T 5053a7afb4994ecdbc92b129c58903cf P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:45:56.288393 5364 tablet_bootstrap.cc:654] T 5053a7afb4994ecdbc92b129c58903cf P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:56.289433 5360 tablet_bootstrap.cc:492] T 253d158b340a444182793f18239bfa22 P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:45:56.290380 5360 tablet_bootstrap.cc:654] T 253d158b340a444182793f18239bfa22 P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:56.290501 5366 tablet_bootstrap.cc:492] T 5053a7afb4994ecdbc92b129c58903cf P 923cfab926e7443f8ffd7b21ad0a9d91: No bootstrap required, opened a new log
I20240502 20:45:56.290601 5366 ts_tablet_manager.cc:1367] T 5053a7afb4994ecdbc92b129c58903cf P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 20:45:56.290701 5362 tablet_bootstrap.cc:492] T 5053a7afb4994ecdbc92b129c58903cf P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:45:56.291456 5360 tablet_bootstrap.cc:492] T 253d158b340a444182793f18239bfa22 P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:45:56.291533 5360 ts_tablet_manager.cc:1367] T 253d158b340a444182793f18239bfa22 P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:56.291616 5362 tablet_bootstrap.cc:654] T 5053a7afb4994ecdbc92b129c58903cf P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:56.291563 5366 raft_consensus.cc:348] T 5053a7afb4994ecdbc92b129c58903cf P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:56.291749 5366 raft_consensus.cc:374] T 5053a7afb4994ecdbc92b129c58903cf P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:56.291797 5366 raft_consensus.cc:729] T 5053a7afb4994ecdbc92b129c58903cf P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: FOLLOWER
I20240502 20:45:56.291944 5366 consensus_queue.cc:260] T 5053a7afb4994ecdbc92b129c58903cf P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:56.291983 5360 raft_consensus.cc:348] T 253d158b340a444182793f18239bfa22 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:45:56.292086 5360 raft_consensus.cc:374] T 253d158b340a444182793f18239bfa22 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:56.292127 5360 raft_consensus.cc:729] T 253d158b340a444182793f18239bfa22 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:45:56.292263 5360 consensus_queue.cc:260] T 253d158b340a444182793f18239bfa22 P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:45:56.292433 5366 ts_tablet_manager.cc:1398] T 5053a7afb4994ecdbc92b129c58903cf P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:45:56.292555 5360 ts_tablet_manager.cc:1398] T 253d158b340a444182793f18239bfa22 P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:56.292646 5366 tablet_bootstrap.cc:492] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:45:56.294000 5366 tablet_bootstrap.cc:654] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:56.294183 5364 tablet_bootstrap.cc:492] T 5053a7afb4994ecdbc92b129c58903cf P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:45:56.294281 5364 ts_tablet_manager.cc:1367] T 5053a7afb4994ecdbc92b129c58903cf P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240502 20:45:56.294849 5364 raft_consensus.cc:348] T 5053a7afb4994ecdbc92b129c58903cf P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:56.294981 5364 raft_consensus.cc:374] T 5053a7afb4994ecdbc92b129c58903cf P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:56.295015 5364 raft_consensus.cc:729] T 5053a7afb4994ecdbc92b129c58903cf P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:45:56.295147 5366 tablet_bootstrap.cc:492] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91: No bootstrap required, opened a new log
I20240502 20:45:56.295154 5364 consensus_queue.cc:260] T 5053a7afb4994ecdbc92b129c58903cf P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:56.295229 5366 ts_tablet_manager.cc:1367] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:45:56.295650 5366 raft_consensus.cc:348] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:45:56.295771 5366 raft_consensus.cc:374] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:56.295814 5366 raft_consensus.cc:729] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: FOLLOWER
I20240502 20:45:56.295964 5366 consensus_queue.cc:260] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:45:56.296312 5366 ts_tablet_manager.cc:1398] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:56.296536 5366 tablet_bootstrap.cc:492] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:45:56.296834 5364 ts_tablet_manager.cc:1398] T 5053a7afb4994ecdbc92b129c58903cf P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 20:45:56.297801 5366 tablet_bootstrap.cc:654] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:56.298830 5366 tablet_bootstrap.cc:492] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91: No bootstrap required, opened a new log
I20240502 20:45:56.298914 5366 ts_tablet_manager.cc:1367] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:56.299247 5364 tablet_bootstrap.cc:492] T 253d158b340a444182793f18239bfa22 P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:45:56.299394 5366 raft_consensus.cc:348] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:56.299490 5366 raft_consensus.cc:374] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:56.299521 5366 raft_consensus.cc:729] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: FOLLOWER
I20240502 20:45:56.299643 5366 consensus_queue.cc:260] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:56.299916 5366 ts_tablet_manager.cc:1398] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:56.300179 5364 tablet_bootstrap.cc:654] T 253d158b340a444182793f18239bfa22 P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:56.301234 5364 tablet_bootstrap.cc:492] T 253d158b340a444182793f18239bfa22 P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:45:56.301333 5364 ts_tablet_manager.cc:1367] T 253d158b340a444182793f18239bfa22 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 20:45:56.301331 5362 tablet_bootstrap.cc:492] T 5053a7afb4994ecdbc92b129c58903cf P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:45:56.301416 5362 ts_tablet_manager.cc:1367] T 5053a7afb4994ecdbc92b129c58903cf P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240502 20:45:56.301788 5364 raft_consensus.cc:348] T 253d158b340a444182793f18239bfa22 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:45:56.301884 5364 raft_consensus.cc:374] T 253d158b340a444182793f18239bfa22 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:56.301919 5364 raft_consensus.cc:729] T 253d158b340a444182793f18239bfa22 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:45:56.301865 5362 raft_consensus.cc:348] T 5053a7afb4994ecdbc92b129c58903cf P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:56.301973 5362 raft_consensus.cc:374] T 5053a7afb4994ecdbc92b129c58903cf P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:56.302004 5362 raft_consensus.cc:729] T 5053a7afb4994ecdbc92b129c58903cf P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:45:56.302122 5362 consensus_queue.cc:260] T 5053a7afb4994ecdbc92b129c58903cf P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:56.302134 5364 consensus_queue.cc:260] T 253d158b340a444182793f18239bfa22 P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:45:56.302428 5364 ts_tablet_manager.cc:1398] T 253d158b340a444182793f18239bfa22 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:45:56.302443 5362 ts_tablet_manager.cc:1398] T 5053a7afb4994ecdbc92b129c58903cf P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:56.417594 5373 raft_consensus.cc:482] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:56.417738 5373 raft_consensus.cc:504] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:56.418128 5373 leader_election.cc:288] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141), c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097)
I20240502 20:45:56.418498 4639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37fab152a8a44868d3160ec3057bb98" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" is_pre_election: true
I20240502 20:45:56.418516 4506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37fab152a8a44868d3160ec3057bb98" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" is_pre_election: true
I20240502 20:45:56.418649 4639 raft_consensus.cc:2454] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 0.
I20240502 20:45:56.418673 4506 raft_consensus.cc:2454] T d37fab152a8a44868d3160ec3057bb98 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 0.
I20240502 20:45:56.418941 4705 leader_election.cc:302] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3d26eafd51547acb780d4aac5ebe7cc, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:45:56.419126 5373 raft_consensus.cc:2781] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:56.419188 5373 raft_consensus.cc:482] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:56.419215 5373 raft_consensus.cc:3037] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:56.420199 5373 raft_consensus.cc:504] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:56.420544 5373 leader_election.cc:288] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 election: Requested vote from peers 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141), c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097)
I20240502 20:45:56.420893 4506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37fab152a8a44868d3160ec3057bb98" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc"
I20240502 20:45:56.420892 4639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37fab152a8a44868d3160ec3057bb98" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91"
I20240502 20:45:56.421041 4506 raft_consensus.cc:3037] T d37fab152a8a44868d3160ec3057bb98 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:56.421041 4639 raft_consensus.cc:3037] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:56.422024 4639 raft_consensus.cc:2454] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 1.
I20240502 20:45:56.422024 4506 raft_consensus.cc:2454] T d37fab152a8a44868d3160ec3057bb98 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 1.
I20240502 20:45:56.422333 4703 leader_election.cc:302] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:45:56.422533 5373 raft_consensus.cc:2781] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:56.422618 5373 raft_consensus.cc:686] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Becoming Leader. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Running, Role: LEADER
I20240502 20:45:56.422791 5373 consensus_queue.cc:237] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:56.424588 4230 catalog_manager.cc:5484] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd reported cstate change: term changed from 0 to 1, leader changed from <none> to ed47c023452a497192f35d2f617bf9bd (127.30.32.68). New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } }
I20240502 20:45:56.434038 5511 raft_consensus.cc:482] T 5053a7afb4994ecdbc92b129c58903cf P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:56.434150 5511 raft_consensus.cc:504] T 5053a7afb4994ecdbc92b129c58903cf P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:56.434607 5511 leader_election.cc:288] T 5053a7afb4994ecdbc92b129c58903cf P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141), ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:45:56.434985 4769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5053a7afb4994ecdbc92b129c58903cf" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd" is_pre_election: true
I20240502 20:45:56.435125 4769 raft_consensus.cc:2454] T 5053a7afb4994ecdbc92b129c58903cf P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 0.
I20240502 20:45:56.435439 4444 leader_election.cc:302] T 5053a7afb4994ecdbc92b129c58903cf P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3d26eafd51547acb780d4aac5ebe7cc, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:45:56.435612 5511 raft_consensus.cc:2781] T 5053a7afb4994ecdbc92b129c58903cf P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:56.435671 5511 raft_consensus.cc:482] T 5053a7afb4994ecdbc92b129c58903cf P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:56.435727 5511 raft_consensus.cc:3037] T 5053a7afb4994ecdbc92b129c58903cf P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:56.436650 5511 raft_consensus.cc:504] T 5053a7afb4994ecdbc92b129c58903cf P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:56.437008 5511 leader_election.cc:288] T 5053a7afb4994ecdbc92b129c58903cf P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 election: Requested vote from peers 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141), ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:45:56.437342 4769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5053a7afb4994ecdbc92b129c58903cf" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd"
I20240502 20:45:56.437453 4769 raft_consensus.cc:3037] T 5053a7afb4994ecdbc92b129c58903cf P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:56.438323 4769 raft_consensus.cc:2454] T 5053a7afb4994ecdbc92b129c58903cf P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 1.
I20240502 20:45:56.438519 4639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5053a7afb4994ecdbc92b129c58903cf" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" is_pre_election: true
I20240502 20:45:56.438583 4444 leader_election.cc:302] T 5053a7afb4994ecdbc92b129c58903cf P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3d26eafd51547acb780d4aac5ebe7cc, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:45:56.438658 4639 raft_consensus.cc:2454] T 5053a7afb4994ecdbc92b129c58903cf P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 0.
I20240502 20:45:56.438678 4638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5053a7afb4994ecdbc92b129c58903cf" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91"
I20240502 20:45:56.438750 4638 raft_consensus.cc:3037] T 5053a7afb4994ecdbc92b129c58903cf P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:56.438750 5511 raft_consensus.cc:2781] T 5053a7afb4994ecdbc92b129c58903cf P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:56.438922 5511 raft_consensus.cc:686] T 5053a7afb4994ecdbc92b129c58903cf P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Becoming Leader. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Running, Role: LEADER
I20240502 20:45:56.439049 5511 consensus_queue.cc:237] T 5053a7afb4994ecdbc92b129c58903cf P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:56.439756 4638 raft_consensus.cc:2454] T 5053a7afb4994ecdbc92b129c58903cf P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 1.
I20240502 20:45:56.440799 4230 catalog_manager.cc:5484] T 5053a7afb4994ecdbc92b129c58903cf P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: term changed from 0 to 1, leader changed from <none> to c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66). New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } }
I20240502 20:45:56.522518 5560 raft_consensus.cc:482] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:56.522680 5560 raft_consensus.cc:504] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:45:56.523125 5560 leader_election.cc:288] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097), c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:45:56.523612 4506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253d158b340a444182793f18239bfa22" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" is_pre_election: true
I20240502 20:45:56.523612 4377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253d158b340a444182793f18239bfa22" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2" is_pre_election: true
I20240502 20:45:56.523752 4506 raft_consensus.cc:2454] T 253d158b340a444182793f18239bfa22 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 0.
I20240502 20:45:56.523766 4377 raft_consensus.cc:2454] T 253d158b340a444182793f18239bfa22 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 0.
I20240502 20:45:56.524104 4574 leader_election.cc:302] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, c3d26eafd51547acb780d4aac5ebe7cc; no voters:
I20240502 20:45:56.524274 5560 raft_consensus.cc:2781] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:56.524338 5560 raft_consensus.cc:482] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:56.524377 5560 raft_consensus.cc:3037] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:56.525269 5560 raft_consensus.cc:504] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:45:56.525588 5560 leader_election.cc:288] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 election: Requested vote from peers c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097), c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:45:56.525954 4506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253d158b340a444182793f18239bfa22" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc"
I20240502 20:45:56.525982 4377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253d158b340a444182793f18239bfa22" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2"
I20240502 20:45:56.526099 4377 raft_consensus.cc:3037] T 253d158b340a444182793f18239bfa22 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:56.526095 4506 raft_consensus.cc:3037] T 253d158b340a444182793f18239bfa22 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:56.526963 4377 raft_consensus.cc:2454] T 253d158b340a444182793f18239bfa22 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 1.
I20240502 20:45:56.527249 4575 leader_election.cc:302] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:45:56.527424 5560 raft_consensus.cc:2781] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:56.527506 5560 raft_consensus.cc:686] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Becoming Leader. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Running, Role: LEADER
I20240502 20:45:56.527637 5560 consensus_queue.cc:237] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:45:56.529363 4230 catalog_manager.cc:5484] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67). New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } }
I20240502 20:45:56.549925 4506 raft_consensus.cc:2454] T 253d158b340a444182793f18239bfa22 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 1.
W20240502 20:45:56.696218 5668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:56.696457 5668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:56.699560 5668 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240502 20:45:56.791514 4377 raft_consensus.cc:1261] T 253d158b340a444182793f18239bfa22 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:56.791675 4506 raft_consensus.cc:1261] T d37fab152a8a44868d3160ec3057bb98 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:56.791951 4769 raft_consensus.cc:1261] T 5053a7afb4994ecdbc92b129c58903cf P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:56.792443 4638 raft_consensus.cc:1261] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:56.792413 5511 consensus_queue.cc:1035] T 5053a7afb4994ecdbc92b129c58903cf P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:56.792819 5380 consensus_queue.cc:1035] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:56.793685 5560 consensus_queue.cc:1035] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:56.794083 5373 consensus_queue.cc:1035] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:56.794682 4638 raft_consensus.cc:1261] T 5053a7afb4994ecdbc92b129c58903cf P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:56.796046 5511 consensus_queue.cc:1035] T 5053a7afb4994ecdbc92b129c58903cf P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:56.799739 4507 raft_consensus.cc:1261] T 253d158b340a444182793f18239bfa22 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:56.800186 5603 consensus_queue.cc:1035] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:56.805534 5715 mvcc.cc:205] Tried to move back new op lower bound from 7023340571813494784 to 7023340570309365760. Current Snapshot: MvccSnapshot[applied={T|T < 7023340571813494784}]
I20240502 20:45:56.812116 5713 mvcc.cc:205] Tried to move back new op lower bound from 7023340571812442112 to 7023340570738847744. Current Snapshot: MvccSnapshot[applied={T|T < 7023340571812442112}]
INSERT report
rows total: 2000
time total: 115.873 ms
time per row: 0.0579367 ms
I20240502 20:45:56.858866 4233 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41812:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:45:56.864706 4486 tablet_service.cc:1467] Processing CreateTablet for tablet 61a6d9a8e21e4a098cfad7281f0e0fd6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=f4c78cd17a604c19aba2b334e324fc04]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:56.864888 4749 tablet_service.cc:1467] Processing CreateTablet for tablet 61a6d9a8e21e4a098cfad7281f0e0fd6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=f4c78cd17a604c19aba2b334e324fc04]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:56.865065 4486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61a6d9a8e21e4a098cfad7281f0e0fd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:56.865258 4749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61a6d9a8e21e4a098cfad7281f0e0fd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:56.866964 5732 tablet_bootstrap.cc:492] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:45:56.867913 5732 tablet_bootstrap.cc:654] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:56.868012 4750 tablet_service.cc:1467] Processing CreateTablet for tablet f6d4a3019ba54eb8b3506b88f04a6262 (DEFAULT_TABLE table=rebalance_test_table_6 [id=f4c78cd17a604c19aba2b334e324fc04]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:56.868240 4750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6d4a3019ba54eb8b3506b88f04a6262. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:56.868750 4486 tablet_service.cc:1467] Processing CreateTablet for tablet f6d4a3019ba54eb8b3506b88f04a6262 (DEFAULT_TABLE table=rebalance_test_table_6 [id=f4c78cd17a604c19aba2b334e324fc04]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:56.868960 4486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6d4a3019ba54eb8b3506b88f04a6262. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:56.870044 4619 tablet_service.cc:1467] Processing CreateTablet for tablet 3ecf52a5823142beaf4217419be1d01c (DEFAULT_TABLE table=rebalance_test_table_6 [id=f4c78cd17a604c19aba2b334e324fc04]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:56.870298 4619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ecf52a5823142beaf4217419be1d01c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:56.872125 5735 tablet_bootstrap.cc:492] T 3ecf52a5823142beaf4217419be1d01c P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:45:56.872707 4356 tablet_service.cc:1467] Processing CreateTablet for tablet 61a6d9a8e21e4a098cfad7281f0e0fd6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=f4c78cd17a604c19aba2b334e324fc04]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:56.872799 4355 tablet_service.cc:1467] Processing CreateTablet for tablet 3ecf52a5823142beaf4217419be1d01c (DEFAULT_TABLE table=rebalance_test_table_6 [id=f4c78cd17a604c19aba2b334e324fc04]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:56.872999 4356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61a6d9a8e21e4a098cfad7281f0e0fd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:56.873055 5735 tablet_bootstrap.cc:654] T 3ecf52a5823142beaf4217419be1d01c P 923cfab926e7443f8ffd7b21ad0a9d91: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:56.873304 4357 tablet_service.cc:1467] Processing CreateTablet for tablet f6d4a3019ba54eb8b3506b88f04a6262 (DEFAULT_TABLE table=rebalance_test_table_6 [id=f4c78cd17a604c19aba2b334e324fc04]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:56.873529 4357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6d4a3019ba54eb8b3506b88f04a6262. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:56.873926 4355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ecf52a5823142beaf4217419be1d01c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:56.874644 4749 tablet_service.cc:1467] Processing CreateTablet for tablet 3ecf52a5823142beaf4217419be1d01c (DEFAULT_TABLE table=rebalance_test_table_6 [id=f4c78cd17a604c19aba2b334e324fc04]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:56.874876 4749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ecf52a5823142beaf4217419be1d01c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:56.874882 5733 tablet_bootstrap.cc:492] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:45:56.875558 5733 tablet_bootstrap.cc:654] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:56.876473 5735 tablet_bootstrap.cc:492] T 3ecf52a5823142beaf4217419be1d01c P 923cfab926e7443f8ffd7b21ad0a9d91: No bootstrap required, opened a new log
I20240502 20:45:56.876564 5735 ts_tablet_manager.cc:1367] T 3ecf52a5823142beaf4217419be1d01c P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 20:45:56.876658 5732 tablet_bootstrap.cc:492] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:45:56.876760 5732 ts_tablet_manager.cc:1367] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240502 20:45:56.877081 5733 tablet_bootstrap.cc:492] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:45:56.877183 5733 ts_tablet_manager.cc:1367] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:45:56.877225 5732 raft_consensus.cc:348] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:56.877337 5732 raft_consensus.cc:374] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:56.877293 5735 raft_consensus.cc:348] T 3ecf52a5823142beaf4217419be1d01c P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:56.877377 5732 raft_consensus.cc:729] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:45:56.877388 5735 raft_consensus.cc:374] T 3ecf52a5823142beaf4217419be1d01c P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:56.877420 5735 raft_consensus.cc:729] T 3ecf52a5823142beaf4217419be1d01c P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: FOLLOWER
I20240502 20:45:56.877496 5732 consensus_queue.cc:260] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:56.877565 5735 consensus_queue.cc:260] T 3ecf52a5823142beaf4217419be1d01c P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:56.877583 5733 raft_consensus.cc:348] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:56.877676 5733 raft_consensus.cc:374] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:56.877714 5733 raft_consensus.cc:729] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:45:56.877781 5732 ts_tablet_manager.cc:1398] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:56.877828 5733 consensus_queue.cc:260] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:56.877873 5735 ts_tablet_manager.cc:1398] T 3ecf52a5823142beaf4217419be1d01c P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:56.877936 5732 tablet_bootstrap.cc:492] T f6d4a3019ba54eb8b3506b88f04a6262 P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:45:56.878152 5733 ts_tablet_manager.cc:1398] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:56.878275 5733 tablet_bootstrap.cc:492] T f6d4a3019ba54eb8b3506b88f04a6262 P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:45:56.878751 5732 tablet_bootstrap.cc:654] T f6d4a3019ba54eb8b3506b88f04a6262 P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:56.879295 5733 tablet_bootstrap.cc:654] T f6d4a3019ba54eb8b3506b88f04a6262 P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:56.880519 5733 tablet_bootstrap.cc:492] T f6d4a3019ba54eb8b3506b88f04a6262 P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:45:56.880574 5732 tablet_bootstrap.cc:492] T f6d4a3019ba54eb8b3506b88f04a6262 P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:45:56.880600 5733 ts_tablet_manager.cc:1367] T f6d4a3019ba54eb8b3506b88f04a6262 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:56.880648 5732 ts_tablet_manager.cc:1367] T f6d4a3019ba54eb8b3506b88f04a6262 P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 20:45:56.881076 5733 raft_consensus.cc:348] T f6d4a3019ba54eb8b3506b88f04a6262 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:56.881098 5732 raft_consensus.cc:348] T f6d4a3019ba54eb8b3506b88f04a6262 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:56.881217 5732 raft_consensus.cc:374] T f6d4a3019ba54eb8b3506b88f04a6262 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:56.881219 5733 raft_consensus.cc:374] T f6d4a3019ba54eb8b3506b88f04a6262 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:56.881254 5733 raft_consensus.cc:729] T f6d4a3019ba54eb8b3506b88f04a6262 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:45:56.881255 5732 raft_consensus.cc:729] T f6d4a3019ba54eb8b3506b88f04a6262 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:45:56.881371 5733 consensus_queue.cc:260] T f6d4a3019ba54eb8b3506b88f04a6262 P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:56.881651 5733 ts_tablet_manager.cc:1398] T f6d4a3019ba54eb8b3506b88f04a6262 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:56.881927 5732 consensus_queue.cc:260] T f6d4a3019ba54eb8b3506b88f04a6262 P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:56.882272 5732 ts_tablet_manager.cc:1398] T f6d4a3019ba54eb8b3506b88f04a6262 P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:45:56.882436 5732 tablet_bootstrap.cc:492] T 3ecf52a5823142beaf4217419be1d01c P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:45:56.883126 5732 tablet_bootstrap.cc:654] T 3ecf52a5823142beaf4217419be1d01c P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:56.884127 5732 tablet_bootstrap.cc:492] T 3ecf52a5823142beaf4217419be1d01c P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:45:56.884205 5732 ts_tablet_manager.cc:1367] T 3ecf52a5823142beaf4217419be1d01c P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:45:56.884603 5732 raft_consensus.cc:348] T 3ecf52a5823142beaf4217419be1d01c P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:56.884714 5732 raft_consensus.cc:374] T 3ecf52a5823142beaf4217419be1d01c P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:56.884760 5732 raft_consensus.cc:729] T 3ecf52a5823142beaf4217419be1d01c P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:45:56.884881 5732 consensus_queue.cc:260] T 3ecf52a5823142beaf4217419be1d01c P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:56.885174 5732 ts_tablet_manager.cc:1398] T 3ecf52a5823142beaf4217419be1d01c P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:56.894953 5741 tablet_bootstrap.cc:492] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:45:56.895845 5741 tablet_bootstrap.cc:654] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:56.896764 5741 tablet_bootstrap.cc:492] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:45:56.896850 5741 ts_tablet_manager.cc:1367] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:45:56.897241 5741 raft_consensus.cc:348] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:56.897339 5741 raft_consensus.cc:374] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:56.897373 5741 raft_consensus.cc:729] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:45:56.897516 5741 consensus_queue.cc:260] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:56.897783 5741 ts_tablet_manager.cc:1398] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:56.897938 5741 tablet_bootstrap.cc:492] T 3ecf52a5823142beaf4217419be1d01c P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:45:56.898653 5741 tablet_bootstrap.cc:654] T 3ecf52a5823142beaf4217419be1d01c P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:56.899847 5741 tablet_bootstrap.cc:492] T 3ecf52a5823142beaf4217419be1d01c P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:45:56.899930 5741 ts_tablet_manager.cc:1367] T 3ecf52a5823142beaf4217419be1d01c P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:45:56.900307 5741 raft_consensus.cc:348] T 3ecf52a5823142beaf4217419be1d01c P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:56.900401 5741 raft_consensus.cc:374] T 3ecf52a5823142beaf4217419be1d01c P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:56.900435 5741 raft_consensus.cc:729] T 3ecf52a5823142beaf4217419be1d01c P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:45:56.900573 5741 consensus_queue.cc:260] T 3ecf52a5823142beaf4217419be1d01c P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:56.900866 5741 ts_tablet_manager.cc:1398] T 3ecf52a5823142beaf4217419be1d01c P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:45:56.901075 5741 tablet_bootstrap.cc:492] T f6d4a3019ba54eb8b3506b88f04a6262 P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:45:56.901929 5741 tablet_bootstrap.cc:654] T f6d4a3019ba54eb8b3506b88f04a6262 P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:56.903028 5741 tablet_bootstrap.cc:492] T f6d4a3019ba54eb8b3506b88f04a6262 P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:45:56.903122 5741 ts_tablet_manager.cc:1367] T f6d4a3019ba54eb8b3506b88f04a6262 P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:45:56.903541 5741 raft_consensus.cc:348] T f6d4a3019ba54eb8b3506b88f04a6262 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:56.903630 5741 raft_consensus.cc:374] T f6d4a3019ba54eb8b3506b88f04a6262 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:56.903650 5741 raft_consensus.cc:729] T f6d4a3019ba54eb8b3506b88f04a6262 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:45:56.903764 5741 consensus_queue.cc:260] T f6d4a3019ba54eb8b3506b88f04a6262 P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:56.904018 5741 ts_tablet_manager.cc:1398] T f6d4a3019ba54eb8b3506b88f04a6262 P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:45:56.979465 5665 raft_consensus.cc:482] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:56.979590 5665 raft_consensus.cc:504] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:56.979983 5665 leader_election.cc:288] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827), ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:45:56.980420 4377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61a6d9a8e21e4a098cfad7281f0e0fd6" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2" is_pre_election: true
I20240502 20:45:56.980444 4769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61a6d9a8e21e4a098cfad7281f0e0fd6" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd" is_pre_election: true
I20240502 20:45:56.980577 4377 raft_consensus.cc:2454] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 0.
I20240502 20:45:56.980577 4769 raft_consensus.cc:2454] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 0.
I20240502 20:45:56.980829 4444 leader_election.cc:302] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3d26eafd51547acb780d4aac5ebe7cc, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:45:56.981006 5665 raft_consensus.cc:2781] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:56.981065 5665 raft_consensus.cc:482] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:56.981112 5665 raft_consensus.cc:3037] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:56.981973 5665 raft_consensus.cc:504] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:56.982323 5665 leader_election.cc:288] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 election: Requested vote from peers c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827), ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:45:56.982626 4377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61a6d9a8e21e4a098cfad7281f0e0fd6" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2"
I20240502 20:45:56.982687 4769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61a6d9a8e21e4a098cfad7281f0e0fd6" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd"
I20240502 20:45:56.982790 4769 raft_consensus.cc:3037] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:56.982787 4377 raft_consensus.cc:3037] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:56.983707 4769 raft_consensus.cc:2454] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 1.
I20240502 20:45:56.983707 4377 raft_consensus.cc:2454] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 1.
I20240502 20:45:56.983989 4444 leader_election.cc:302] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3d26eafd51547acb780d4aac5ebe7cc, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:45:56.984146 5665 raft_consensus.cc:2781] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:56.984241 5665 raft_consensus.cc:686] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Becoming Leader. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Running, Role: LEADER
I20240502 20:45:56.984366 5665 consensus_queue.cc:237] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:56.986380 4233 catalog_manager.cc:5484] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: term changed from 0 to 1, leader changed from <none> to c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66). New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:56.997115 5545 raft_consensus.cc:482] T f6d4a3019ba54eb8b3506b88f04a6262 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:56.997228 5545 raft_consensus.cc:504] T f6d4a3019ba54eb8b3506b88f04a6262 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:56.997591 5545 leader_election.cc:288] T f6d4a3019ba54eb8b3506b88f04a6262 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193), c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097)
I20240502 20:45:56.997925 4769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d4a3019ba54eb8b3506b88f04a6262" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd" is_pre_election: true
I20240502 20:45:56.997938 4507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d4a3019ba54eb8b3506b88f04a6262" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" is_pre_election: true
I20240502 20:45:56.998042 4507 raft_consensus.cc:2454] T f6d4a3019ba54eb8b3506b88f04a6262 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 0.
I20240502 20:45:56.998056 4769 raft_consensus.cc:2454] T f6d4a3019ba54eb8b3506b88f04a6262 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 0.
I20240502 20:45:56.998299 4312 leader_election.cc:302] T f6d4a3019ba54eb8b3506b88f04a6262 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3d26eafd51547acb780d4aac5ebe7cc, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:45:56.998453 5545 raft_consensus.cc:2781] T f6d4a3019ba54eb8b3506b88f04a6262 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:56.998512 5545 raft_consensus.cc:482] T f6d4a3019ba54eb8b3506b88f04a6262 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:56.998565 5545 raft_consensus.cc:3037] T f6d4a3019ba54eb8b3506b88f04a6262 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:56.999369 5545 raft_consensus.cc:504] T f6d4a3019ba54eb8b3506b88f04a6262 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:56.999684 5545 leader_election.cc:288] T f6d4a3019ba54eb8b3506b88f04a6262 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 election: Requested vote from peers ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193), c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097)
I20240502 20:45:57.000013 4769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d4a3019ba54eb8b3506b88f04a6262" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd"
I20240502 20:45:57.000013 4507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d4a3019ba54eb8b3506b88f04a6262" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc"
I20240502 20:45:57.000128 4507 raft_consensus.cc:3037] T f6d4a3019ba54eb8b3506b88f04a6262 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:57.000128 4769 raft_consensus.cc:3037] T f6d4a3019ba54eb8b3506b88f04a6262 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:57.001042 4507 raft_consensus.cc:2454] T f6d4a3019ba54eb8b3506b88f04a6262 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 1.
I20240502 20:45:57.001042 4769 raft_consensus.cc:2454] T f6d4a3019ba54eb8b3506b88f04a6262 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 1.
I20240502 20:45:57.001333 4313 leader_election.cc:302] T f6d4a3019ba54eb8b3506b88f04a6262 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:45:57.001509 5545 raft_consensus.cc:2781] T f6d4a3019ba54eb8b3506b88f04a6262 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:57.001722 5545 raft_consensus.cc:686] T f6d4a3019ba54eb8b3506b88f04a6262 P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEADER]: Becoming Leader. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Running, Role: LEADER
I20240502 20:45:57.001899 5545 consensus_queue.cc:237] T f6d4a3019ba54eb8b3506b88f04a6262 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:57.003540 4233 catalog_manager.cc:5484] T f6d4a3019ba54eb8b3506b88f04a6262 P c92e6e46782849479c8a0c6d82276fb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65). New cstate: current_term: 1 leader_uuid: "c92e6e46782849479c8a0c6d82276fb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:57.107769 5380 raft_consensus.cc:482] T 3ecf52a5823142beaf4217419be1d01c P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:57.107896 5380 raft_consensus.cc:504] T 3ecf52a5823142beaf4217419be1d01c P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:57.108296 5380 leader_election.cc:288] T 3ecf52a5823142beaf4217419be1d01c P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827), 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:45:57.108669 4377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecf52a5823142beaf4217419be1d01c" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2" is_pre_election: true
I20240502 20:45:57.108829 4377 raft_consensus.cc:2454] T 3ecf52a5823142beaf4217419be1d01c P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 0.
I20240502 20:45:57.108887 4639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecf52a5823142beaf4217419be1d01c" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" is_pre_election: true
I20240502 20:45:57.109077 4639 raft_consensus.cc:2454] T 3ecf52a5823142beaf4217419be1d01c P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 0.
I20240502 20:45:57.109138 4706 leader_election.cc:302] T 3ecf52a5823142beaf4217419be1d01c P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:45:57.109323 5380 raft_consensus.cc:2781] T 3ecf52a5823142beaf4217419be1d01c P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:57.109392 5380 raft_consensus.cc:482] T 3ecf52a5823142beaf4217419be1d01c P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:57.109437 5380 raft_consensus.cc:3037] T 3ecf52a5823142beaf4217419be1d01c P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:57.110297 5380 raft_consensus.cc:504] T 3ecf52a5823142beaf4217419be1d01c P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:57.110646 5380 leader_election.cc:288] T 3ecf52a5823142beaf4217419be1d01c P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 election: Requested vote from peers c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827), 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:45:57.110992 4377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecf52a5823142beaf4217419be1d01c" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2"
I20240502 20:45:57.111033 4639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecf52a5823142beaf4217419be1d01c" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91"
I20240502 20:45:57.111148 4639 raft_consensus.cc:3037] T 3ecf52a5823142beaf4217419be1d01c P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:57.111148 4377 raft_consensus.cc:3037] T 3ecf52a5823142beaf4217419be1d01c P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:57.112092 4377 raft_consensus.cc:2454] T 3ecf52a5823142beaf4217419be1d01c P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 1.
I20240502 20:45:57.112092 4639 raft_consensus.cc:2454] T 3ecf52a5823142beaf4217419be1d01c P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 1.
I20240502 20:45:57.112397 4706 leader_election.cc:302] T 3ecf52a5823142beaf4217419be1d01c P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:45:57.112666 5380 raft_consensus.cc:2781] T 3ecf52a5823142beaf4217419be1d01c P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:57.112749 5380 raft_consensus.cc:686] T 3ecf52a5823142beaf4217419be1d01c P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Becoming Leader. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Running, Role: LEADER
I20240502 20:45:57.112885 5380 consensus_queue.cc:237] T 3ecf52a5823142beaf4217419be1d01c P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:57.114823 4233 catalog_manager.cc:5484] T 3ecf52a5823142beaf4217419be1d01c P ed47c023452a497192f35d2f617bf9bd reported cstate change: term changed from 0 to 1, leader changed from <none> to ed47c023452a497192f35d2f617bf9bd (127.30.32.68). New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } }
W20240502 20:45:57.272054 5749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:57.272303 5749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:57.275569 5749 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240502 20:45:57.365615 4769 raft_consensus.cc:1261] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:57.366348 5511 consensus_queue.cc:1035] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:57.369345 4770 raft_consensus.cc:1261] T f6d4a3019ba54eb8b3506b88f04a6262 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:57.370015 4507 raft_consensus.cc:1261] T f6d4a3019ba54eb8b3506b88f04a6262 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:57.370007 5545 consensus_queue.cc:1035] T f6d4a3019ba54eb8b3506b88f04a6262 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:57.370728 5545 consensus_queue.cc:1035] T f6d4a3019ba54eb8b3506b88f04a6262 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:57.373960 4377 raft_consensus.cc:1261] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:57.375229 5511 consensus_queue.cc:1035] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:57.377305 4375 raft_consensus.cc:1261] T 3ecf52a5823142beaf4217419be1d01c P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:57.378162 5380 consensus_queue.cc:1035] T 3ecf52a5823142beaf4217419be1d01c P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:57.381536 4639 raft_consensus.cc:1261] T 3ecf52a5823142beaf4217419be1d01c P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:57.382227 5380 consensus_queue.cc:1035] T 3ecf52a5823142beaf4217419be1d01c P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 133.453 ms
time per row: 0.0667263 ms
I20240502 20:45:57.451965 4230 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41812:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:45:57.457933 4619 tablet_service.cc:1467] Processing CreateTablet for tablet d37730b109d04b6395f75c7552524877 (DEFAULT_TABLE table=rebalance_test_table_7 [id=1e32a0fff569475cbd44b12c4fd5e8e6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:57.458045 4357 tablet_service.cc:1467] Processing CreateTablet for tablet d37730b109d04b6395f75c7552524877 (DEFAULT_TABLE table=rebalance_test_table_7 [id=1e32a0fff569475cbd44b12c4fd5e8e6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:57.458307 4619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d37730b109d04b6395f75c7552524877. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:57.458379 4357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d37730b109d04b6395f75c7552524877. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:57.458494 4618 tablet_service.cc:1467] Processing CreateTablet for tablet d6582234196c4fe4a82b3182787700de (DEFAULT_TABLE table=rebalance_test_table_7 [id=1e32a0fff569475cbd44b12c4fd5e8e6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:57.458742 4618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6582234196c4fe4a82b3182787700de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:57.458840 4356 tablet_service.cc:1467] Processing CreateTablet for tablet 428ffbac75db466191cb3bef63b4eb79 (DEFAULT_TABLE table=rebalance_test_table_7 [id=1e32a0fff569475cbd44b12c4fd5e8e6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:57.459045 4356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 428ffbac75db466191cb3bef63b4eb79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:57.460893 4617 tablet_service.cc:1467] Processing CreateTablet for tablet 428ffbac75db466191cb3bef63b4eb79 (DEFAULT_TABLE table=rebalance_test_table_7 [id=1e32a0fff569475cbd44b12c4fd5e8e6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:57.461177 4617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 428ffbac75db466191cb3bef63b4eb79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:57.461375 5811 tablet_bootstrap.cc:492] T d37730b109d04b6395f75c7552524877 P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:45:57.461396 4749 tablet_service.cc:1467] Processing CreateTablet for tablet 428ffbac75db466191cb3bef63b4eb79 (DEFAULT_TABLE table=rebalance_test_table_7 [id=1e32a0fff569475cbd44b12c4fd5e8e6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:57.461445 4750 tablet_service.cc:1467] Processing CreateTablet for tablet d6582234196c4fe4a82b3182787700de (DEFAULT_TABLE table=rebalance_test_table_7 [id=1e32a0fff569475cbd44b12c4fd5e8e6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:57.461658 4749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 428ffbac75db466191cb3bef63b4eb79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:57.461841 4750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6582234196c4fe4a82b3182787700de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:57.462363 5811 tablet_bootstrap.cc:654] T d37730b109d04b6395f75c7552524877 P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:57.463052 5812 tablet_bootstrap.cc:492] T d6582234196c4fe4a82b3182787700de P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:45:57.463904 5812 tablet_bootstrap.cc:654] T d6582234196c4fe4a82b3182787700de P 923cfab926e7443f8ffd7b21ad0a9d91: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:57.464295 4486 tablet_service.cc:1467] Processing CreateTablet for tablet d37730b109d04b6395f75c7552524877 (DEFAULT_TABLE table=rebalance_test_table_7 [id=1e32a0fff569475cbd44b12c4fd5e8e6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:57.464596 4486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d37730b109d04b6395f75c7552524877. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:57.464679 5814 tablet_bootstrap.cc:492] T 428ffbac75db466191cb3bef63b4eb79 P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:45:57.464810 5811 tablet_bootstrap.cc:492] T d37730b109d04b6395f75c7552524877 P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:45:57.464906 5811 ts_tablet_manager.cc:1367] T d37730b109d04b6395f75c7552524877 P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 20:45:57.465353 5811 raft_consensus.cc:348] T d37730b109d04b6395f75c7552524877 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:57.465441 5814 tablet_bootstrap.cc:654] T 428ffbac75db466191cb3bef63b4eb79 P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:57.465469 5811 raft_consensus.cc:374] T d37730b109d04b6395f75c7552524877 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:57.465505 5811 raft_consensus.cc:729] T d37730b109d04b6395f75c7552524877 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:45:57.465664 5811 consensus_queue.cc:260] T d37730b109d04b6395f75c7552524877 P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:57.465975 5811 ts_tablet_manager.cc:1398] T d37730b109d04b6395f75c7552524877 P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:57.466162 5811 tablet_bootstrap.cc:492] T 428ffbac75db466191cb3bef63b4eb79 P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:45:57.464295 4488 tablet_service.cc:1467] Processing CreateTablet for tablet d6582234196c4fe4a82b3182787700de (DEFAULT_TABLE table=rebalance_test_table_7 [id=1e32a0fff569475cbd44b12c4fd5e8e6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:57.467064 4488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6582234196c4fe4a82b3182787700de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:57.467283 5811 tablet_bootstrap.cc:654] T 428ffbac75db466191cb3bef63b4eb79 P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:57.467938 5812 tablet_bootstrap.cc:492] T d6582234196c4fe4a82b3182787700de P 923cfab926e7443f8ffd7b21ad0a9d91: No bootstrap required, opened a new log
I20240502 20:45:57.467998 5812 ts_tablet_manager.cc:1367] T d6582234196c4fe4a82b3182787700de P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240502 20:45:57.468353 5812 raft_consensus.cc:348] T d6582234196c4fe4a82b3182787700de P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:57.468442 5812 raft_consensus.cc:374] T d6582234196c4fe4a82b3182787700de P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:57.468475 5812 raft_consensus.cc:729] T d6582234196c4fe4a82b3182787700de P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: FOLLOWER
I20240502 20:45:57.468616 5812 consensus_queue.cc:260] T d6582234196c4fe4a82b3182787700de P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:57.467054 5817 tablet_bootstrap.cc:492] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:45:57.469189 5814 tablet_bootstrap.cc:492] T 428ffbac75db466191cb3bef63b4eb79 P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:45:57.469525 5814 ts_tablet_manager.cc:1367] T 428ffbac75db466191cb3bef63b4eb79 P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 20:45:57.470225 5817 tablet_bootstrap.cc:654] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:57.470214 5814 raft_consensus.cc:348] T 428ffbac75db466191cb3bef63b4eb79 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:57.470408 5814 raft_consensus.cc:374] T 428ffbac75db466191cb3bef63b4eb79 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:57.470280 5812 ts_tablet_manager.cc:1398] T d6582234196c4fe4a82b3182787700de P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:45:57.470464 5814 raft_consensus.cc:729] T 428ffbac75db466191cb3bef63b4eb79 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:45:57.470598 5814 consensus_queue.cc:260] T 428ffbac75db466191cb3bef63b4eb79 P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:57.470638 5812 tablet_bootstrap.cc:492] T d37730b109d04b6395f75c7552524877 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:45:57.471012 5814 ts_tablet_manager.cc:1398] T 428ffbac75db466191cb3bef63b4eb79 P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:57.471176 5814 tablet_bootstrap.cc:492] T d6582234196c4fe4a82b3182787700de P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:45:57.471318 5817 tablet_bootstrap.cc:492] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:45:57.471396 5817 ts_tablet_manager.cc:1367] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 20:45:57.471799 5817 raft_consensus.cc:348] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:57.471884 5812 tablet_bootstrap.cc:654] T d37730b109d04b6395f75c7552524877 P 923cfab926e7443f8ffd7b21ad0a9d91: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:57.471922 5817 raft_consensus.cc:374] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:57.471956 5817 raft_consensus.cc:729] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:45:57.472108 5817 consensus_queue.cc:260] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:57.473007 5817 ts_tablet_manager.cc:1398] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:45:57.473166 5817 tablet_bootstrap.cc:492] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:45:57.473760 5811 tablet_bootstrap.cc:492] T 428ffbac75db466191cb3bef63b4eb79 P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:45:57.473764 5812 tablet_bootstrap.cc:492] T d37730b109d04b6395f75c7552524877 P 923cfab926e7443f8ffd7b21ad0a9d91: No bootstrap required, opened a new log
I20240502 20:45:57.473841 5811 ts_tablet_manager.cc:1367] T 428ffbac75db466191cb3bef63b4eb79 P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 20:45:57.473841 5812 ts_tablet_manager.cc:1367] T d37730b109d04b6395f75c7552524877 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:45:57.474202 5812 raft_consensus.cc:348] T d37730b109d04b6395f75c7552524877 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:57.474306 5812 raft_consensus.cc:374] T d37730b109d04b6395f75c7552524877 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:57.474337 5812 raft_consensus.cc:729] T d37730b109d04b6395f75c7552524877 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: FOLLOWER
I20240502 20:45:57.474282 5811 raft_consensus.cc:348] T 428ffbac75db466191cb3bef63b4eb79 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:57.474375 5811 raft_consensus.cc:374] T 428ffbac75db466191cb3bef63b4eb79 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:57.474409 5811 raft_consensus.cc:729] T 428ffbac75db466191cb3bef63b4eb79 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:45:57.474491 5812 consensus_queue.cc:260] T d37730b109d04b6395f75c7552524877 P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:57.474516 5811 consensus_queue.cc:260] T 428ffbac75db466191cb3bef63b4eb79 P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:57.474800 5811 ts_tablet_manager.cc:1398] T 428ffbac75db466191cb3bef63b4eb79 P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:57.475379 5812 ts_tablet_manager.cc:1398] T d37730b109d04b6395f75c7552524877 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:57.475890 5812 tablet_bootstrap.cc:492] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:45:57.476296 5814 tablet_bootstrap.cc:654] T d6582234196c4fe4a82b3182787700de P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:57.477402 5814 tablet_bootstrap.cc:492] T d6582234196c4fe4a82b3182787700de P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:45:57.477488 5814 ts_tablet_manager.cc:1367] T d6582234196c4fe4a82b3182787700de P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 20:45:57.477821 5817 tablet_bootstrap.cc:654] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:57.477882 5812 tablet_bootstrap.cc:654] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:57.477963 5814 raft_consensus.cc:348] T d6582234196c4fe4a82b3182787700de P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:57.478081 5814 raft_consensus.cc:374] T d6582234196c4fe4a82b3182787700de P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:57.478127 5814 raft_consensus.cc:729] T d6582234196c4fe4a82b3182787700de P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:45:57.478427 5814 consensus_queue.cc:260] T d6582234196c4fe4a82b3182787700de P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:57.478799 5814 ts_tablet_manager.cc:1398] T d6582234196c4fe4a82b3182787700de P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:57.480012 5812 tablet_bootstrap.cc:492] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91: No bootstrap required, opened a new log
I20240502 20:45:57.480106 5812 ts_tablet_manager.cc:1367] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 20:45:57.480533 5812 raft_consensus.cc:348] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:57.480625 5812 raft_consensus.cc:374] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:57.480624 5817 tablet_bootstrap.cc:492] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:45:57.480662 5812 raft_consensus.cc:729] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: FOLLOWER
I20240502 20:45:57.480706 5817 ts_tablet_manager.cc:1367] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 20:45:57.480804 5812 consensus_queue.cc:260] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:57.481117 5812 ts_tablet_manager.cc:1398] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:57.481345 5817 raft_consensus.cc:348] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:57.481456 5817 raft_consensus.cc:374] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:57.481489 5817 raft_consensus.cc:729] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:45:57.481631 5817 consensus_queue.cc:260] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:57.481912 5817 ts_tablet_manager.cc:1398] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:45:57.518214 5511 raft_consensus.cc:482] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:57.518365 5511 raft_consensus.cc:504] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:57.518770 5511 leader_election.cc:288] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141), ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:45:57.519228 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6582234196c4fe4a82b3182787700de" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd" is_pre_election: true
I20240502 20:45:57.519228 4639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6582234196c4fe4a82b3182787700de" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" is_pre_election: true
I20240502 20:45:57.519382 4770 raft_consensus.cc:2454] T d6582234196c4fe4a82b3182787700de P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 0.
I20240502 20:45:57.519382 4639 raft_consensus.cc:2454] T d6582234196c4fe4a82b3182787700de P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 0.
I20240502 20:45:57.519706 4444 leader_election.cc:302] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3d26eafd51547acb780d4aac5ebe7cc, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:45:57.519876 5511 raft_consensus.cc:2781] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:57.519935 5511 raft_consensus.cc:482] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:57.519984 5511 raft_consensus.cc:3037] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:57.520810 5511 raft_consensus.cc:504] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:57.521183 5511 leader_election.cc:288] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 election: Requested vote from peers 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141), ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:45:57.521503 4639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6582234196c4fe4a82b3182787700de" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91"
I20240502 20:45:57.521559 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6582234196c4fe4a82b3182787700de" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd"
I20240502 20:45:57.521643 4639 raft_consensus.cc:3037] T d6582234196c4fe4a82b3182787700de P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:57.521683 4770 raft_consensus.cc:3037] T d6582234196c4fe4a82b3182787700de P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:57.522495 4639 raft_consensus.cc:2454] T d6582234196c4fe4a82b3182787700de P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 1.
I20240502 20:45:57.522540 4770 raft_consensus.cc:2454] T d6582234196c4fe4a82b3182787700de P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 1.
I20240502 20:45:57.522807 4441 leader_election.cc:302] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, c3d26eafd51547acb780d4aac5ebe7cc; no voters:
I20240502 20:45:57.522966 5511 raft_consensus.cc:2781] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:57.523056 5511 raft_consensus.cc:686] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Becoming Leader. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Running, Role: LEADER
I20240502 20:45:57.523213 5511 consensus_queue.cc:237] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:57.524859 4230 catalog_manager.cc:5484] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: term changed from 0 to 1, leader changed from <none> to c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66). New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:57.595898 5728 raft_consensus.cc:482] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:57.596020 5728 raft_consensus.cc:504] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:57.596428 5728 leader_election.cc:288] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193), c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:45:57.596874 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "428ffbac75db466191cb3bef63b4eb79" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd" is_pre_election: true
I20240502 20:45:57.596889 4377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "428ffbac75db466191cb3bef63b4eb79" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2" is_pre_election: true
I20240502 20:45:57.597048 4377 raft_consensus.cc:2454] T 428ffbac75db466191cb3bef63b4eb79 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 0.
I20240502 20:45:57.597050 4770 raft_consensus.cc:2454] T 428ffbac75db466191cb3bef63b4eb79 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 0.
I20240502 20:45:57.597368 4575 leader_election.cc:302] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:45:57.597559 5728 raft_consensus.cc:2781] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:57.597623 5728 raft_consensus.cc:482] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:57.597652 5728 raft_consensus.cc:3037] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:57.598453 5728 raft_consensus.cc:504] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:57.598769 5728 leader_election.cc:288] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 election: Requested vote from peers ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193), c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:45:57.599129 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "428ffbac75db466191cb3bef63b4eb79" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd"
I20240502 20:45:57.599225 4770 raft_consensus.cc:3037] T 428ffbac75db466191cb3bef63b4eb79 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:57.599187 4377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "428ffbac75db466191cb3bef63b4eb79" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2"
I20240502 20:45:57.599318 4377 raft_consensus.cc:3037] T 428ffbac75db466191cb3bef63b4eb79 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:57.599987 4770 raft_consensus.cc:2454] T 428ffbac75db466191cb3bef63b4eb79 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 1.
I20240502 20:45:57.600135 4377 raft_consensus.cc:2454] T 428ffbac75db466191cb3bef63b4eb79 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 1.
I20240502 20:45:57.600302 4575 leader_election.cc:302] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:45:57.600483 5728 raft_consensus.cc:2781] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:57.600747 5728 raft_consensus.cc:686] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Becoming Leader. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Running, Role: LEADER
I20240502 20:45:57.600899 5728 consensus_queue.cc:237] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:57.603678 4230 catalog_manager.cc:5484] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67). New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:57.627125 5511 raft_consensus.cc:482] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:57.627283 5511 raft_consensus.cc:504] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:57.627688 5511 leader_election.cc:288] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141), c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:45:57.628217 4639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37730b109d04b6395f75c7552524877" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" is_pre_election: true
I20240502 20:45:57.628225 4377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37730b109d04b6395f75c7552524877" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2" is_pre_election: true
I20240502 20:45:57.628379 4639 raft_consensus.cc:2454] T d37730b109d04b6395f75c7552524877 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 0.
I20240502 20:45:57.628379 4377 raft_consensus.cc:2454] T d37730b109d04b6395f75c7552524877 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 0.
I20240502 20:45:57.628703 4444 leader_election.cc:302] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3d26eafd51547acb780d4aac5ebe7cc, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:45:57.628906 5511 raft_consensus.cc:2781] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:57.628975 5511 raft_consensus.cc:482] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:57.629050 5511 raft_consensus.cc:3037] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:57.629923 5511 raft_consensus.cc:504] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:57.630308 5511 leader_election.cc:288] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 election: Requested vote from peers 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141), c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:45:57.630669 4639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37730b109d04b6395f75c7552524877" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91"
I20240502 20:45:57.630677 4377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37730b109d04b6395f75c7552524877" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2"
I20240502 20:45:57.630784 4377 raft_consensus.cc:3037] T d37730b109d04b6395f75c7552524877 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:57.630796 4639 raft_consensus.cc:3037] T d37730b109d04b6395f75c7552524877 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:57.631628 4377 raft_consensus.cc:2454] T d37730b109d04b6395f75c7552524877 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 1.
I20240502 20:45:57.631664 4639 raft_consensus.cc:2454] T d37730b109d04b6395f75c7552524877 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 1.
I20240502 20:45:57.632032 4441 leader_election.cc:302] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, c3d26eafd51547acb780d4aac5ebe7cc; no voters:
I20240502 20:45:57.632254 5511 raft_consensus.cc:2781] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:57.632346 5511 raft_consensus.cc:686] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Becoming Leader. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Running, Role: LEADER
I20240502 20:45:57.632488 5511 consensus_queue.cc:237] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:57.634447 4230 catalog_manager.cc:5484] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: term changed from 0 to 1, leader changed from <none> to c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66). New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } }
W20240502 20:45:57.759644 5828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:57.759919 5828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:57.762959 5828 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240502 20:45:57.846304 4770 raft_consensus.cc:1261] T 428ffbac75db466191cb3bef63b4eb79 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:57.846902 5728 consensus_queue.cc:1035] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:57.847460 4377 raft_consensus.cc:1261] T d37730b109d04b6395f75c7552524877 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:57.847921 5511 consensus_queue.cc:1035] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:57.849550 4768 raft_consensus.cc:1261] T d6582234196c4fe4a82b3182787700de P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:57.849537 4639 raft_consensus.cc:1261] T d6582234196c4fe4a82b3182787700de P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:57.850090 5665 consensus_queue.cc:1035] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:57.856101 5665 consensus_queue.cc:1035] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:57.856258 4375 raft_consensus.cc:1261] T 428ffbac75db466191cb3bef63b4eb79 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:57.856638 5855 consensus_queue.cc:1035] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:57.867668 4638 raft_consensus.cc:1261] T d37730b109d04b6395f75c7552524877 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:57.868252 5511 consensus_queue.cc:1035] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 112.261 ms
time per row: 0.0561304 ms
I20240502 20:45:57.917739 4235 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41812:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:45:57.923568 4356 tablet_service.cc:1467] Processing CreateTablet for tablet 28029d4e66d1453cbde9811ffc2a3283 (DEFAULT_TABLE table=rebalance_test_table_8 [id=e3df92c02b8746f28ae1f38db2a1630e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:57.923902 4356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28029d4e66d1453cbde9811ffc2a3283. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:57.924239 4488 tablet_service.cc:1467] Processing CreateTablet for tablet 28029d4e66d1453cbde9811ffc2a3283 (DEFAULT_TABLE table=rebalance_test_table_8 [id=e3df92c02b8746f28ae1f38db2a1630e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:57.924399 4355 tablet_service.cc:1467] Processing CreateTablet for tablet 02bddc1866e04858894e047ec100a6f0 (DEFAULT_TABLE table=rebalance_test_table_8 [id=e3df92c02b8746f28ae1f38db2a1630e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:57.924510 4488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28029d4e66d1453cbde9811ffc2a3283. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:57.924613 4355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02bddc1866e04858894e047ec100a6f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:57.926096 5811 tablet_bootstrap.cc:492] T 28029d4e66d1453cbde9811ffc2a3283 P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:45:57.926313 4617 tablet_service.cc:1467] Processing CreateTablet for tablet 28029d4e66d1453cbde9811ffc2a3283 (DEFAULT_TABLE table=rebalance_test_table_8 [id=e3df92c02b8746f28ae1f38db2a1630e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:57.926569 4617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28029d4e66d1453cbde9811ffc2a3283. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:57.926761 4619 tablet_service.cc:1467] Processing CreateTablet for tablet 02bddc1866e04858894e047ec100a6f0 (DEFAULT_TABLE table=rebalance_test_table_8 [id=e3df92c02b8746f28ae1f38db2a1630e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:57.926978 4619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02bddc1866e04858894e047ec100a6f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:57.927043 5811 tablet_bootstrap.cc:654] T 28029d4e66d1453cbde9811ffc2a3283 P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:57.927937 5817 tablet_bootstrap.cc:492] T 28029d4e66d1453cbde9811ffc2a3283 P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:45:57.928109 5811 tablet_bootstrap.cc:492] T 28029d4e66d1453cbde9811ffc2a3283 P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:45:57.928180 5811 ts_tablet_manager.cc:1367] T 28029d4e66d1453cbde9811ffc2a3283 P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:57.928601 5812 tablet_bootstrap.cc:492] T 28029d4e66d1453cbde9811ffc2a3283 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:45:57.928588 5811 raft_consensus.cc:348] T 28029d4e66d1453cbde9811ffc2a3283 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:45:57.928711 5811 raft_consensus.cc:374] T 28029d4e66d1453cbde9811ffc2a3283 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:57.928732 5817 tablet_bootstrap.cc:654] T 28029d4e66d1453cbde9811ffc2a3283 P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:57.928745 5811 raft_consensus.cc:729] T 28029d4e66d1453cbde9811ffc2a3283 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:45:57.928879 5811 consensus_queue.cc:260] T 28029d4e66d1453cbde9811ffc2a3283 P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:45:57.929230 5811 ts_tablet_manager.cc:1398] T 28029d4e66d1453cbde9811ffc2a3283 P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:57.929407 5812 tablet_bootstrap.cc:654] T 28029d4e66d1453cbde9811ffc2a3283 P 923cfab926e7443f8ffd7b21ad0a9d91: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:57.929440 5811 tablet_bootstrap.cc:492] T 02bddc1866e04858894e047ec100a6f0 P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:45:57.929845 4357 tablet_service.cc:1467] Processing CreateTablet for tablet 47c5087093214fbeae35590b4b3c3d42 (DEFAULT_TABLE table=rebalance_test_table_8 [id=e3df92c02b8746f28ae1f38db2a1630e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:57.930089 4357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47c5087093214fbeae35590b4b3c3d42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:57.930287 5811 tablet_bootstrap.cc:654] T 02bddc1866e04858894e047ec100a6f0 P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:57.931089 5812 tablet_bootstrap.cc:492] T 28029d4e66d1453cbde9811ffc2a3283 P 923cfab926e7443f8ffd7b21ad0a9d91: No bootstrap required, opened a new log
I20240502 20:45:57.931165 5812 ts_tablet_manager.cc:1367] T 28029d4e66d1453cbde9811ffc2a3283 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240502 20:45:57.931563 5812 raft_consensus.cc:348] T 28029d4e66d1453cbde9811ffc2a3283 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:45:57.931636 5812 raft_consensus.cc:374] T 28029d4e66d1453cbde9811ffc2a3283 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:57.931656 5812 raft_consensus.cc:729] T 28029d4e66d1453cbde9811ffc2a3283 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: FOLLOWER
I20240502 20:45:57.931756 5812 consensus_queue.cc:260] T 28029d4e66d1453cbde9811ffc2a3283 P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:45:57.932018 5812 ts_tablet_manager.cc:1398] T 28029d4e66d1453cbde9811ffc2a3283 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:45:57.932176 5812 tablet_bootstrap.cc:492] T 02bddc1866e04858894e047ec100a6f0 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:45:57.932420 4750 tablet_service.cc:1467] Processing CreateTablet for tablet 47c5087093214fbeae35590b4b3c3d42 (DEFAULT_TABLE table=rebalance_test_table_8 [id=e3df92c02b8746f28ae1f38db2a1630e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:57.932709 4750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47c5087093214fbeae35590b4b3c3d42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:57.932885 5812 tablet_bootstrap.cc:654] T 02bddc1866e04858894e047ec100a6f0 P 923cfab926e7443f8ffd7b21ad0a9d91: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:57.934219 5811 tablet_bootstrap.cc:492] T 02bddc1866e04858894e047ec100a6f0 P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:45:57.934302 5811 ts_tablet_manager.cc:1367] T 02bddc1866e04858894e047ec100a6f0 P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 20:45:57.934736 5811 raft_consensus.cc:348] T 02bddc1866e04858894e047ec100a6f0 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:57.934798 5814 tablet_bootstrap.cc:492] T 47c5087093214fbeae35590b4b3c3d42 P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:45:57.934835 5811 raft_consensus.cc:374] T 02bddc1866e04858894e047ec100a6f0 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:57.934868 5811 raft_consensus.cc:729] T 02bddc1866e04858894e047ec100a6f0 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:45:57.935045 4749 tablet_service.cc:1467] Processing CreateTablet for tablet 02bddc1866e04858894e047ec100a6f0 (DEFAULT_TABLE table=rebalance_test_table_8 [id=e3df92c02b8746f28ae1f38db2a1630e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:57.935278 4749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02bddc1866e04858894e047ec100a6f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:57.935554 5814 tablet_bootstrap.cc:654] T 47c5087093214fbeae35590b4b3c3d42 P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:57.936226 5811 consensus_queue.cc:260] T 02bddc1866e04858894e047ec100a6f0 P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:57.936405 5814 tablet_bootstrap.cc:492] T 47c5087093214fbeae35590b4b3c3d42 P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:45:57.936481 5814 ts_tablet_manager.cc:1367] T 47c5087093214fbeae35590b4b3c3d42 P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:45:57.936797 5814 raft_consensus.cc:348] T 47c5087093214fbeae35590b4b3c3d42 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:57.936868 5814 raft_consensus.cc:374] T 47c5087093214fbeae35590b4b3c3d42 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:57.936888 5814 raft_consensus.cc:729] T 47c5087093214fbeae35590b4b3c3d42 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:45:57.937011 5814 consensus_queue.cc:260] T 47c5087093214fbeae35590b4b3c3d42 P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:57.937273 5814 ts_tablet_manager.cc:1398] T 47c5087093214fbeae35590b4b3c3d42 P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:57.937424 5814 tablet_bootstrap.cc:492] T 02bddc1866e04858894e047ec100a6f0 P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:45:57.937522 5811 ts_tablet_manager.cc:1398] T 02bddc1866e04858894e047ec100a6f0 P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 20:45:57.937691 5811 tablet_bootstrap.cc:492] T 47c5087093214fbeae35590b4b3c3d42 P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:45:57.938275 5814 tablet_bootstrap.cc:654] T 02bddc1866e04858894e047ec100a6f0 P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:57.938671 5811 tablet_bootstrap.cc:654] T 47c5087093214fbeae35590b4b3c3d42 P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:57.939770 5814 tablet_bootstrap.cc:492] T 02bddc1866e04858894e047ec100a6f0 P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:45:57.939898 5814 ts_tablet_manager.cc:1367] T 02bddc1866e04858894e047ec100a6f0 P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:45:57.940387 5814 raft_consensus.cc:348] T 02bddc1866e04858894e047ec100a6f0 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:57.940477 5814 raft_consensus.cc:374] T 02bddc1866e04858894e047ec100a6f0 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:57.940511 5814 raft_consensus.cc:729] T 02bddc1866e04858894e047ec100a6f0 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:45:57.940649 5814 consensus_queue.cc:260] T 02bddc1866e04858894e047ec100a6f0 P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:57.940869 4488 tablet_service.cc:1467] Processing CreateTablet for tablet 47c5087093214fbeae35590b4b3c3d42 (DEFAULT_TABLE table=rebalance_test_table_8 [id=e3df92c02b8746f28ae1f38db2a1630e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:57.940923 5814 ts_tablet_manager.cc:1398] T 02bddc1866e04858894e047ec100a6f0 P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:57.941138 4488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47c5087093214fbeae35590b4b3c3d42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:57.941776 5817 tablet_bootstrap.cc:492] T 28029d4e66d1453cbde9811ffc2a3283 P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:45:57.941879 5817 ts_tablet_manager.cc:1367] T 28029d4e66d1453cbde9811ffc2a3283 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240502 20:45:57.942351 5817 raft_consensus.cc:348] T 28029d4e66d1453cbde9811ffc2a3283 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:45:57.942463 5817 raft_consensus.cc:374] T 28029d4e66d1453cbde9811ffc2a3283 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:57.942502 5817 raft_consensus.cc:729] T 28029d4e66d1453cbde9811ffc2a3283 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:45:57.942641 5817 consensus_queue.cc:260] T 28029d4e66d1453cbde9811ffc2a3283 P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:45:57.942940 5817 ts_tablet_manager.cc:1398] T 28029d4e66d1453cbde9811ffc2a3283 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:57.944082 5817 tablet_bootstrap.cc:492] T 47c5087093214fbeae35590b4b3c3d42 P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:45:57.945027 5812 tablet_bootstrap.cc:492] T 02bddc1866e04858894e047ec100a6f0 P 923cfab926e7443f8ffd7b21ad0a9d91: No bootstrap required, opened a new log
I20240502 20:45:57.945135 5812 ts_tablet_manager.cc:1367] T 02bddc1866e04858894e047ec100a6f0 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.000s
I20240502 20:45:57.945199 5811 tablet_bootstrap.cc:492] T 47c5087093214fbeae35590b4b3c3d42 P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:45:57.945286 5811 ts_tablet_manager.cc:1367] T 47c5087093214fbeae35590b4b3c3d42 P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 20:45:57.945609 5812 raft_consensus.cc:348] T 02bddc1866e04858894e047ec100a6f0 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:57.945659 5811 raft_consensus.cc:348] T 47c5087093214fbeae35590b4b3c3d42 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:57.945735 5812 raft_consensus.cc:374] T 02bddc1866e04858894e047ec100a6f0 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:57.945755 5811 raft_consensus.cc:374] T 47c5087093214fbeae35590b4b3c3d42 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:57.945780 5812 raft_consensus.cc:729] T 02bddc1866e04858894e047ec100a6f0 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: FOLLOWER
I20240502 20:45:57.945792 5811 raft_consensus.cc:729] T 47c5087093214fbeae35590b4b3c3d42 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:45:57.945709 5817 tablet_bootstrap.cc:654] T 47c5087093214fbeae35590b4b3c3d42 P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:57.945906 5811 consensus_queue.cc:260] T 47c5087093214fbeae35590b4b3c3d42 P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:57.946193 5811 ts_tablet_manager.cc:1398] T 47c5087093214fbeae35590b4b3c3d42 P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:57.947126 5812 consensus_queue.cc:260] T 02bddc1866e04858894e047ec100a6f0 P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:57.948223 5817 tablet_bootstrap.cc:492] T 47c5087093214fbeae35590b4b3c3d42 P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:45:57.948343 5817 ts_tablet_manager.cc:1367] T 47c5087093214fbeae35590b4b3c3d42 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 20:45:57.948817 5817 raft_consensus.cc:348] T 47c5087093214fbeae35590b4b3c3d42 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:57.948923 5817 raft_consensus.cc:374] T 47c5087093214fbeae35590b4b3c3d42 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:57.948969 5817 raft_consensus.cc:729] T 47c5087093214fbeae35590b4b3c3d42 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:45:57.949012 5812 ts_tablet_manager.cc:1398] T 02bddc1866e04858894e047ec100a6f0 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 20:45:57.949110 5817 consensus_queue.cc:260] T 47c5087093214fbeae35590b4b3c3d42 P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:57.949440 5817 ts_tablet_manager.cc:1398] T 47c5087093214fbeae35590b4b3c3d42 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:45:57.962229 5373 raft_consensus.cc:482] T 02bddc1866e04858894e047ec100a6f0 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:57.962327 5373 raft_consensus.cc:504] T 02bddc1866e04858894e047ec100a6f0 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:57.962675 5373 leader_election.cc:288] T 02bddc1866e04858894e047ec100a6f0 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141), c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:45:57.962978 4639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02bddc1866e04858894e047ec100a6f0" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" is_pre_election: true
I20240502 20:45:57.963017 4377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02bddc1866e04858894e047ec100a6f0" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2" is_pre_election: true
I20240502 20:45:57.963099 4639 raft_consensus.cc:2454] T 02bddc1866e04858894e047ec100a6f0 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 0.
I20240502 20:45:57.963126 4377 raft_consensus.cc:2454] T 02bddc1866e04858894e047ec100a6f0 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 0.
I20240502 20:45:57.963376 4706 leader_election.cc:302] T 02bddc1866e04858894e047ec100a6f0 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:45:57.963529 5373 raft_consensus.cc:2781] T 02bddc1866e04858894e047ec100a6f0 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:57.963605 5373 raft_consensus.cc:482] T 02bddc1866e04858894e047ec100a6f0 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:57.963646 5373 raft_consensus.cc:3037] T 02bddc1866e04858894e047ec100a6f0 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:57.964381 5373 raft_consensus.cc:504] T 02bddc1866e04858894e047ec100a6f0 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:57.964715 5373 leader_election.cc:288] T 02bddc1866e04858894e047ec100a6f0 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 election: Requested vote from peers 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141), c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:45:57.965020 4639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02bddc1866e04858894e047ec100a6f0" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91"
I20240502 20:45:57.965037 4377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02bddc1866e04858894e047ec100a6f0" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2"
I20240502 20:45:57.965130 4377 raft_consensus.cc:3037] T 02bddc1866e04858894e047ec100a6f0 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:57.965128 4639 raft_consensus.cc:3037] T 02bddc1866e04858894e047ec100a6f0 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:57.965826 4639 raft_consensus.cc:2454] T 02bddc1866e04858894e047ec100a6f0 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 1.
I20240502 20:45:57.965881 4377 raft_consensus.cc:2454] T 02bddc1866e04858894e047ec100a6f0 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 1.
I20240502 20:45:57.966073 4703 leader_election.cc:302] T 02bddc1866e04858894e047ec100a6f0 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:45:57.966231 5373 raft_consensus.cc:2781] T 02bddc1866e04858894e047ec100a6f0 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:57.966322 5373 raft_consensus.cc:686] T 02bddc1866e04858894e047ec100a6f0 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Becoming Leader. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Running, Role: LEADER
I20240502 20:45:57.966486 5373 consensus_queue.cc:237] T 02bddc1866e04858894e047ec100a6f0 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:57.968009 4235 catalog_manager.cc:5484] T 02bddc1866e04858894e047ec100a6f0 P ed47c023452a497192f35d2f617bf9bd reported cstate change: term changed from 0 to 1, leader changed from <none> to ed47c023452a497192f35d2f617bf9bd (127.30.32.68). New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:58.054597 5810 raft_consensus.cc:482] T 47c5087093214fbeae35590b4b3c3d42 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:58.054718 5810 raft_consensus.cc:504] T 47c5087093214fbeae35590b4b3c3d42 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:58.055065 5810 leader_election.cc:288] T 47c5087093214fbeae35590b4b3c3d42 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193), c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097)
I20240502 20:45:58.055413 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c5087093214fbeae35590b4b3c3d42" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd" is_pre_election: true
I20240502 20:45:58.055449 4506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c5087093214fbeae35590b4b3c3d42" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" is_pre_election: true
I20240502 20:45:58.055538 4768 raft_consensus.cc:2454] T 47c5087093214fbeae35590b4b3c3d42 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 0.
I20240502 20:45:58.055575 4506 raft_consensus.cc:2454] T 47c5087093214fbeae35590b4b3c3d42 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 0.
I20240502 20:45:58.055863 4313 leader_election.cc:302] T 47c5087093214fbeae35590b4b3c3d42 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:45:58.056037 5810 raft_consensus.cc:2781] T 47c5087093214fbeae35590b4b3c3d42 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:58.056098 5810 raft_consensus.cc:482] T 47c5087093214fbeae35590b4b3c3d42 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:58.056152 5810 raft_consensus.cc:3037] T 47c5087093214fbeae35590b4b3c3d42 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:58.057009 5810 raft_consensus.cc:504] T 47c5087093214fbeae35590b4b3c3d42 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:58.057370 5810 leader_election.cc:288] T 47c5087093214fbeae35590b4b3c3d42 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 election: Requested vote from peers ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193), c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097)
I20240502 20:45:58.057838 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c5087093214fbeae35590b4b3c3d42" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd"
I20240502 20:45:58.057914 4506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c5087093214fbeae35590b4b3c3d42" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc"
I20240502 20:45:58.057961 4768 raft_consensus.cc:3037] T 47c5087093214fbeae35590b4b3c3d42 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:58.058036 4506 raft_consensus.cc:3037] T 47c5087093214fbeae35590b4b3c3d42 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:58.058755 4768 raft_consensus.cc:2454] T 47c5087093214fbeae35590b4b3c3d42 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 1.
I20240502 20:45:58.058873 4506 raft_consensus.cc:2454] T 47c5087093214fbeae35590b4b3c3d42 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 1.
I20240502 20:45:58.059031 4313 leader_election.cc:302] T 47c5087093214fbeae35590b4b3c3d42 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:45:58.059201 5810 raft_consensus.cc:2781] T 47c5087093214fbeae35590b4b3c3d42 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:58.059289 5810 raft_consensus.cc:686] T 47c5087093214fbeae35590b4b3c3d42 P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEADER]: Becoming Leader. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Running, Role: LEADER
I20240502 20:45:58.059608 5810 consensus_queue.cc:237] T 47c5087093214fbeae35590b4b3c3d42 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:58.060307 5864 raft_consensus.cc:482] T 28029d4e66d1453cbde9811ffc2a3283 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:58.060387 5864 raft_consensus.cc:504] T 28029d4e66d1453cbde9811ffc2a3283 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:45:58.060765 5864 leader_election.cc:288] T 28029d4e66d1453cbde9811ffc2a3283 P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827), 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:45:58.061049 4377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28029d4e66d1453cbde9811ffc2a3283" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2" is_pre_election: true
I20240502 20:45:58.061161 4377 raft_consensus.cc:2454] T 28029d4e66d1453cbde9811ffc2a3283 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 0.
I20240502 20:45:58.061218 4639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28029d4e66d1453cbde9811ffc2a3283" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" is_pre_election: true
I20240502 20:45:58.061350 4639 raft_consensus.cc:2454] T 28029d4e66d1453cbde9811ffc2a3283 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 0.
I20240502 20:45:58.061609 4444 leader_election.cc:302] T 28029d4e66d1453cbde9811ffc2a3283 P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3d26eafd51547acb780d4aac5ebe7cc, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:45:58.061580 4235 catalog_manager.cc:5484] T 47c5087093214fbeae35590b4b3c3d42 P c92e6e46782849479c8a0c6d82276fb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65). New cstate: current_term: 1 leader_uuid: "c92e6e46782849479c8a0c6d82276fb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:58.061769 5864 raft_consensus.cc:2781] T 28029d4e66d1453cbde9811ffc2a3283 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:58.061825 5864 raft_consensus.cc:482] T 28029d4e66d1453cbde9811ffc2a3283 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:58.061867 5864 raft_consensus.cc:3037] T 28029d4e66d1453cbde9811ffc2a3283 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:58.062707 5864 raft_consensus.cc:504] T 28029d4e66d1453cbde9811ffc2a3283 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:45:58.063053 5864 leader_election.cc:288] T 28029d4e66d1453cbde9811ffc2a3283 P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 election: Requested vote from peers c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827), 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:45:58.063395 4639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28029d4e66d1453cbde9811ffc2a3283" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91"
I20240502 20:45:58.063467 4377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28029d4e66d1453cbde9811ffc2a3283" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2"
I20240502 20:45:58.063517 4639 raft_consensus.cc:3037] T 28029d4e66d1453cbde9811ffc2a3283 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:58.063602 4377 raft_consensus.cc:3037] T 28029d4e66d1453cbde9811ffc2a3283 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:58.064265 4639 raft_consensus.cc:2454] T 28029d4e66d1453cbde9811ffc2a3283 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 1.
I20240502 20:45:58.064462 4377 raft_consensus.cc:2454] T 28029d4e66d1453cbde9811ffc2a3283 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 1.
I20240502 20:45:58.064553 4441 leader_election.cc:302] T 28029d4e66d1453cbde9811ffc2a3283 P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, c3d26eafd51547acb780d4aac5ebe7cc; no voters:
I20240502 20:45:58.064831 5864 raft_consensus.cc:2781] T 28029d4e66d1453cbde9811ffc2a3283 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:58.064918 5864 raft_consensus.cc:686] T 28029d4e66d1453cbde9811ffc2a3283 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Becoming Leader. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Running, Role: LEADER
I20240502 20:45:58.065069 5864 consensus_queue.cc:237] T 28029d4e66d1453cbde9811ffc2a3283 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:45:58.066673 4235 catalog_manager.cc:5484] T 28029d4e66d1453cbde9811ffc2a3283 P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: term changed from 0 to 1, leader changed from <none> to c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66). New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: UNKNOWN } } }
W20240502 20:45:58.204129 5878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:58.204444 5878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:58.207656 5878 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240502 20:45:58.298308 4639 raft_consensus.cc:1261] T 28029d4e66d1453cbde9811ffc2a3283 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:58.298717 5864 consensus_queue.cc:1035] T 28029d4e66d1453cbde9811ffc2a3283 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:58.299907 4506 raft_consensus.cc:1261] T 47c5087093214fbeae35590b4b3c3d42 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:58.300372 5810 consensus_queue.cc:1035] T 47c5087093214fbeae35590b4b3c3d42 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:58.300833 4377 raft_consensus.cc:1261] T 28029d4e66d1453cbde9811ffc2a3283 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:58.303560 4377 raft_consensus.cc:1261] T 02bddc1866e04858894e047ec100a6f0 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:58.304097 5373 consensus_queue.cc:1035] T 02bddc1866e04858894e047ec100a6f0 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:58.305290 5665 consensus_queue.cc:1035] T 28029d4e66d1453cbde9811ffc2a3283 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:58.310333 4768 raft_consensus.cc:1261] T 47c5087093214fbeae35590b4b3c3d42 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:58.310844 5545 consensus_queue.cc:1035] T 47c5087093214fbeae35590b4b3c3d42 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:58.325050 4639 raft_consensus.cc:1261] T 02bddc1866e04858894e047ec100a6f0 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:58.325414 5373 consensus_queue.cc:1035] T 02bddc1866e04858894e047ec100a6f0 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 116.734 ms
time per row: 0.0583668 ms
I20240502 20:45:58.366269 4235 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41812:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:45:58.371711 4619 tablet_service.cc:1467] Processing CreateTablet for tablet d6fb394fdcff436886fe0634d79c024c (DEFAULT_TABLE table=rebalance_test_table_9 [id=7c90eeb05db84afb8f78ab7f14e0ced4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:58.372036 4619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6fb394fdcff436886fe0634d79c024c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:58.372157 4486 tablet_service.cc:1467] Processing CreateTablet for tablet ed6386b6777c4d50b29ff8cd49894eba (DEFAULT_TABLE table=rebalance_test_table_9 [id=7c90eeb05db84afb8f78ab7f14e0ced4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:58.372460 4486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed6386b6777c4d50b29ff8cd49894eba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:58.372642 4487 tablet_service.cc:1467] Processing CreateTablet for tablet c55ba788f592448d8941ff94a8cefc52 (DEFAULT_TABLE table=rebalance_test_table_9 [id=7c90eeb05db84afb8f78ab7f14e0ced4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:58.372864 4487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c55ba788f592448d8941ff94a8cefc52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:58.374063 5812 tablet_bootstrap.cc:492] T d6fb394fdcff436886fe0634d79c024c P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:45:58.374267 4488 tablet_service.cc:1467] Processing CreateTablet for tablet d6fb394fdcff436886fe0634d79c024c (DEFAULT_TABLE table=rebalance_test_table_9 [id=7c90eeb05db84afb8f78ab7f14e0ced4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:58.374476 4488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6fb394fdcff436886fe0634d79c024c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:58.375020 5812 tablet_bootstrap.cc:654] T d6fb394fdcff436886fe0634d79c024c P 923cfab926e7443f8ffd7b21ad0a9d91: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:58.376102 5812 tablet_bootstrap.cc:492] T d6fb394fdcff436886fe0634d79c024c P 923cfab926e7443f8ffd7b21ad0a9d91: No bootstrap required, opened a new log
I20240502 20:45:58.376205 5812 ts_tablet_manager.cc:1367] T d6fb394fdcff436886fe0634d79c024c P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:58.376492 5817 tablet_bootstrap.cc:492] T c55ba788f592448d8941ff94a8cefc52 P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:45:58.376662 5812 raft_consensus.cc:348] T d6fb394fdcff436886fe0634d79c024c P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:58.376780 5812 raft_consensus.cc:374] T d6fb394fdcff436886fe0634d79c024c P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:58.376819 5812 raft_consensus.cc:729] T d6fb394fdcff436886fe0634d79c024c P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: FOLLOWER
I20240502 20:45:58.376948 5812 consensus_queue.cc:260] T d6fb394fdcff436886fe0634d79c024c P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:58.377256 5812 ts_tablet_manager.cc:1398] T d6fb394fdcff436886fe0634d79c024c P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:58.377362 5817 tablet_bootstrap.cc:654] T c55ba788f592448d8941ff94a8cefc52 P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:58.378468 4357 tablet_service.cc:1467] Processing CreateTablet for tablet c55ba788f592448d8941ff94a8cefc52 (DEFAULT_TABLE table=rebalance_test_table_9 [id=7c90eeb05db84afb8f78ab7f14e0ced4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:58.378530 4749 tablet_service.cc:1467] Processing CreateTablet for tablet d6fb394fdcff436886fe0634d79c024c (DEFAULT_TABLE table=rebalance_test_table_9 [id=7c90eeb05db84afb8f78ab7f14e0ced4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:58.378753 4357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c55ba788f592448d8941ff94a8cefc52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:58.378839 4749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6fb394fdcff436886fe0634d79c024c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:58.381621 4750 tablet_service.cc:1467] Processing CreateTablet for tablet ed6386b6777c4d50b29ff8cd49894eba (DEFAULT_TABLE table=rebalance_test_table_9 [id=7c90eeb05db84afb8f78ab7f14e0ced4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:58.381834 4750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed6386b6777c4d50b29ff8cd49894eba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:58.383339 5814 tablet_bootstrap.cc:492] T ed6386b6777c4d50b29ff8cd49894eba P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:45:58.383436 5811 tablet_bootstrap.cc:492] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:45:58.384173 5814 tablet_bootstrap.cc:654] T ed6386b6777c4d50b29ff8cd49894eba P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:58.384258 5811 tablet_bootstrap.cc:654] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:58.385362 5814 tablet_bootstrap.cc:492] T ed6386b6777c4d50b29ff8cd49894eba P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:45:58.385438 5814 ts_tablet_manager.cc:1367] T ed6386b6777c4d50b29ff8cd49894eba P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:58.385627 4617 tablet_service.cc:1467] Processing CreateTablet for tablet c55ba788f592448d8941ff94a8cefc52 (DEFAULT_TABLE table=rebalance_test_table_9 [id=7c90eeb05db84afb8f78ab7f14e0ced4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:58.385627 4619 tablet_service.cc:1467] Processing CreateTablet for tablet ed6386b6777c4d50b29ff8cd49894eba (DEFAULT_TABLE table=rebalance_test_table_9 [id=7c90eeb05db84afb8f78ab7f14e0ced4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:58.385843 5814 raft_consensus.cc:348] T ed6386b6777c4d50b29ff8cd49894eba P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:58.385944 5814 raft_consensus.cc:374] T ed6386b6777c4d50b29ff8cd49894eba P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:58.385959 4617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c55ba788f592448d8941ff94a8cefc52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:58.385978 5814 raft_consensus.cc:729] T ed6386b6777c4d50b29ff8cd49894eba P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:45:58.386111 4619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed6386b6777c4d50b29ff8cd49894eba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:58.386083 5814 consensus_queue.cc:260] T ed6386b6777c4d50b29ff8cd49894eba P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:58.386336 5814 ts_tablet_manager.cc:1398] T ed6386b6777c4d50b29ff8cd49894eba P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:45:58.388144 5812 tablet_bootstrap.cc:492] T c55ba788f592448d8941ff94a8cefc52 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:45:58.388772 5811 tablet_bootstrap.cc:492] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:45:58.388878 5811 ts_tablet_manager.cc:1367] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240502 20:45:58.389045 5812 tablet_bootstrap.cc:654] T c55ba788f592448d8941ff94a8cefc52 P 923cfab926e7443f8ffd7b21ad0a9d91: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:58.389442 5811 raft_consensus.cc:348] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:58.389575 5811 raft_consensus.cc:374] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:58.389613 5811 raft_consensus.cc:729] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:45:58.389750 5811 consensus_queue.cc:260] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:58.390076 5811 ts_tablet_manager.cc:1398] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240502 20:45:58.390218 5812 tablet_bootstrap.cc:492] T c55ba788f592448d8941ff94a8cefc52 P 923cfab926e7443f8ffd7b21ad0a9d91: No bootstrap required, opened a new log
I20240502 20:45:58.390308 5812 ts_tablet_manager.cc:1367] T c55ba788f592448d8941ff94a8cefc52 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 20:45:58.390760 5812 raft_consensus.cc:348] T c55ba788f592448d8941ff94a8cefc52 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:58.390866 5812 raft_consensus.cc:374] T c55ba788f592448d8941ff94a8cefc52 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:58.390897 5812 raft_consensus.cc:729] T c55ba788f592448d8941ff94a8cefc52 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: FOLLOWER
I20240502 20:45:58.390934 5817 tablet_bootstrap.cc:492] T c55ba788f592448d8941ff94a8cefc52 P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:45:58.391029 5817 ts_tablet_manager.cc:1367] T c55ba788f592448d8941ff94a8cefc52 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.002s
I20240502 20:45:58.391017 5812 consensus_queue.cc:260] T c55ba788f592448d8941ff94a8cefc52 P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:58.391283 5812 ts_tablet_manager.cc:1398] T c55ba788f592448d8941ff94a8cefc52 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:45:58.391454 5812 tablet_bootstrap.cc:492] T ed6386b6777c4d50b29ff8cd49894eba P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:45:58.391511 5817 raft_consensus.cc:348] T c55ba788f592448d8941ff94a8cefc52 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:58.391624 5817 raft_consensus.cc:374] T c55ba788f592448d8941ff94a8cefc52 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:58.391662 5817 raft_consensus.cc:729] T c55ba788f592448d8941ff94a8cefc52 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:45:58.391803 5817 consensus_queue.cc:260] T c55ba788f592448d8941ff94a8cefc52 P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:58.392143 5817 ts_tablet_manager.cc:1398] T c55ba788f592448d8941ff94a8cefc52 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:58.392328 5817 tablet_bootstrap.cc:492] T ed6386b6777c4d50b29ff8cd49894eba P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:45:58.392530 5812 tablet_bootstrap.cc:654] T ed6386b6777c4d50b29ff8cd49894eba P 923cfab926e7443f8ffd7b21ad0a9d91: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:58.393966 5814 tablet_bootstrap.cc:492] T d6fb394fdcff436886fe0634d79c024c P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:45:58.394930 5814 tablet_bootstrap.cc:654] T d6fb394fdcff436886fe0634d79c024c P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:58.396045 5814 tablet_bootstrap.cc:492] T d6fb394fdcff436886fe0634d79c024c P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:45:58.396195 5814 ts_tablet_manager.cc:1367] T d6fb394fdcff436886fe0634d79c024c P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:58.396667 5817 tablet_bootstrap.cc:654] T ed6386b6777c4d50b29ff8cd49894eba P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:58.396663 5814 raft_consensus.cc:348] T d6fb394fdcff436886fe0634d79c024c P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:58.396879 5814 raft_consensus.cc:374] T d6fb394fdcff436886fe0634d79c024c P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:58.396971 5814 raft_consensus.cc:729] T d6fb394fdcff436886fe0634d79c024c P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:45:58.397194 5814 consensus_queue.cc:260] T d6fb394fdcff436886fe0634d79c024c P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:58.397548 5814 ts_tablet_manager.cc:1398] T d6fb394fdcff436886fe0634d79c024c P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:58.397673 5817 tablet_bootstrap.cc:492] T ed6386b6777c4d50b29ff8cd49894eba P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:45:58.397763 5817 ts_tablet_manager.cc:1367] T ed6386b6777c4d50b29ff8cd49894eba P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 20:45:58.398195 5817 raft_consensus.cc:348] T ed6386b6777c4d50b29ff8cd49894eba P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:58.398322 5817 raft_consensus.cc:374] T ed6386b6777c4d50b29ff8cd49894eba P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:58.398361 5817 raft_consensus.cc:729] T ed6386b6777c4d50b29ff8cd49894eba P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:45:58.398479 5817 consensus_queue.cc:260] T ed6386b6777c4d50b29ff8cd49894eba P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:58.398736 5817 ts_tablet_manager.cc:1398] T ed6386b6777c4d50b29ff8cd49894eba P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:58.398908 5817 tablet_bootstrap.cc:492] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:45:58.399804 5817 tablet_bootstrap.cc:654] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:58.402725 5812 tablet_bootstrap.cc:492] T ed6386b6777c4d50b29ff8cd49894eba P 923cfab926e7443f8ffd7b21ad0a9d91: No bootstrap required, opened a new log
I20240502 20:45:58.402835 5812 ts_tablet_manager.cc:1367] T ed6386b6777c4d50b29ff8cd49894eba P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240502 20:45:58.402850 5817 tablet_bootstrap.cc:492] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:45:58.402915 5817 ts_tablet_manager.cc:1367] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 20:45:58.403297 5817 raft_consensus.cc:348] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:58.403321 5812 raft_consensus.cc:348] T ed6386b6777c4d50b29ff8cd49894eba P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:58.403409 5812 raft_consensus.cc:374] T ed6386b6777c4d50b29ff8cd49894eba P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:58.403407 5817 raft_consensus.cc:374] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:58.403445 5812 raft_consensus.cc:729] T ed6386b6777c4d50b29ff8cd49894eba P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: FOLLOWER
I20240502 20:45:58.403445 5817 raft_consensus.cc:729] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:45:58.403558 5817 consensus_queue.cc:260] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:58.403558 5812 consensus_queue.cc:260] T ed6386b6777c4d50b29ff8cd49894eba P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:58.403812 5817 ts_tablet_manager.cc:1398] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:45:58.403841 5812 ts_tablet_manager.cc:1398] T ed6386b6777c4d50b29ff8cd49894eba P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:58.462718 5915 raft_consensus.cc:482] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:58.462859 5915 raft_consensus.cc:504] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:58.463287 5915 leader_election.cc:288] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097), 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:45:58.463621 4506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c55ba788f592448d8941ff94a8cefc52" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" is_pre_election: true
I20240502 20:45:58.463656 4639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c55ba788f592448d8941ff94a8cefc52" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" is_pre_election: true
I20240502 20:45:58.463764 4506 raft_consensus.cc:2454] T c55ba788f592448d8941ff94a8cefc52 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 0.
I20240502 20:45:58.463788 4639 raft_consensus.cc:2454] T c55ba788f592448d8941ff94a8cefc52 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 0.
I20240502 20:45:58.464054 4312 leader_election.cc:302] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3d26eafd51547acb780d4aac5ebe7cc, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:45:58.464217 5915 raft_consensus.cc:2781] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:58.464284 5915 raft_consensus.cc:482] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:58.464461 5915 raft_consensus.cc:3037] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:58.465392 5915 raft_consensus.cc:504] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:58.465723 5915 leader_election.cc:288] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 election: Requested vote from peers c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097), 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:45:58.466049 4506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c55ba788f592448d8941ff94a8cefc52" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc"
I20240502 20:45:58.466092 4639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c55ba788f592448d8941ff94a8cefc52" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91"
I20240502 20:45:58.466161 4506 raft_consensus.cc:3037] T c55ba788f592448d8941ff94a8cefc52 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:58.466202 4639 raft_consensus.cc:3037] T c55ba788f592448d8941ff94a8cefc52 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:58.466943 4639 raft_consensus.cc:2454] T c55ba788f592448d8941ff94a8cefc52 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 1.
I20240502 20:45:58.467043 4506 raft_consensus.cc:2454] T c55ba788f592448d8941ff94a8cefc52 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 1.
I20240502 20:45:58.467504 4310 leader_election.cc:302] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:45:58.467684 5915 raft_consensus.cc:2781] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:58.467761 5915 raft_consensus.cc:686] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEADER]: Becoming Leader. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Running, Role: LEADER
I20240502 20:45:58.467901 5915 consensus_queue.cc:237] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:45:58.470083 4235 catalog_manager.cc:5484] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65). New cstate: current_term: 1 leader_uuid: "c92e6e46782849479c8a0c6d82276fb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: HEALTHY } } }
I20240502 20:45:58.565623 5864 raft_consensus.cc:482] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:58.565768 5864 raft_consensus.cc:504] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:58.566198 5864 leader_election.cc:288] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141), ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:45:58.566627 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6fb394fdcff436886fe0634d79c024c" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd" is_pre_election: true
I20240502 20:45:58.566643 4639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6fb394fdcff436886fe0634d79c024c" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" is_pre_election: true
I20240502 20:45:58.566790 4639 raft_consensus.cc:2454] T d6fb394fdcff436886fe0634d79c024c P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 0.
I20240502 20:45:58.566793 4768 raft_consensus.cc:2454] T d6fb394fdcff436886fe0634d79c024c P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 0.
I20240502 20:45:58.567119 4441 leader_election.cc:302] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, c3d26eafd51547acb780d4aac5ebe7cc; no voters:
I20240502 20:45:58.567283 5864 raft_consensus.cc:2781] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:58.567354 5864 raft_consensus.cc:482] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:58.567410 5864 raft_consensus.cc:3037] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:58.568461 5864 raft_consensus.cc:504] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:58.568851 5864 leader_election.cc:288] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 election: Requested vote from peers 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141), ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:45:58.569214 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6fb394fdcff436886fe0634d79c024c" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd"
I20240502 20:45:58.569229 4639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6fb394fdcff436886fe0634d79c024c" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91"
I20240502 20:45:58.569339 4639 raft_consensus.cc:3037] T d6fb394fdcff436886fe0634d79c024c P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:58.569342 4768 raft_consensus.cc:3037] T d6fb394fdcff436886fe0634d79c024c P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:58.570108 4768 raft_consensus.cc:2454] T d6fb394fdcff436886fe0634d79c024c P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 1.
I20240502 20:45:58.570295 4639 raft_consensus.cc:2454] T d6fb394fdcff436886fe0634d79c024c P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 1.
I20240502 20:45:58.570592 4441 leader_election.cc:302] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, c3d26eafd51547acb780d4aac5ebe7cc; no voters:
I20240502 20:45:58.570783 5864 raft_consensus.cc:2781] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:58.570892 5864 raft_consensus.cc:686] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Becoming Leader. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Running, Role: LEADER
I20240502 20:45:58.571035 5864 consensus_queue.cc:237] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:58.572850 4235 catalog_manager.cc:5484] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: term changed from 0 to 1, leader changed from <none> to c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66). New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:58.775446 5825 raft_consensus.cc:482] T ed6386b6777c4d50b29ff8cd49894eba P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:58.775575 5825 raft_consensus.cc:504] T ed6386b6777c4d50b29ff8cd49894eba P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:58.776031 5825 leader_election.cc:288] T ed6386b6777c4d50b29ff8cd49894eba P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097), ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:45:58.776333 4506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed6386b6777c4d50b29ff8cd49894eba" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" is_pre_election: true
I20240502 20:45:58.776490 4506 raft_consensus.cc:2454] T ed6386b6777c4d50b29ff8cd49894eba P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 0.
I20240502 20:45:58.776473 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed6386b6777c4d50b29ff8cd49894eba" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd" is_pre_election: true
I20240502 20:45:58.776624 4768 raft_consensus.cc:2454] T ed6386b6777c4d50b29ff8cd49894eba P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 0.
I20240502 20:45:58.776772 4574 leader_election.cc:302] T ed6386b6777c4d50b29ff8cd49894eba P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, c3d26eafd51547acb780d4aac5ebe7cc; no voters:
I20240502 20:45:58.776947 5825 raft_consensus.cc:2781] T ed6386b6777c4d50b29ff8cd49894eba P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:58.777025 5825 raft_consensus.cc:482] T ed6386b6777c4d50b29ff8cd49894eba P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:58.777079 5825 raft_consensus.cc:3037] T ed6386b6777c4d50b29ff8cd49894eba P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:58.777886 5825 raft_consensus.cc:504] T ed6386b6777c4d50b29ff8cd49894eba P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:58.778208 5825 leader_election.cc:288] T ed6386b6777c4d50b29ff8cd49894eba P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 election: Requested vote from peers c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097), ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:45:58.778481 4507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed6386b6777c4d50b29ff8cd49894eba" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc"
I20240502 20:45:58.778535 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed6386b6777c4d50b29ff8cd49894eba" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd"
I20240502 20:45:58.778605 4507 raft_consensus.cc:3037] T ed6386b6777c4d50b29ff8cd49894eba P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:58.778656 4768 raft_consensus.cc:3037] T ed6386b6777c4d50b29ff8cd49894eba P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:58.779364 4507 raft_consensus.cc:2454] T ed6386b6777c4d50b29ff8cd49894eba P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 1.
I20240502 20:45:58.779369 4768 raft_consensus.cc:2454] T ed6386b6777c4d50b29ff8cd49894eba P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 1.
I20240502 20:45:58.779629 4574 leader_election.cc:302] T ed6386b6777c4d50b29ff8cd49894eba P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, c3d26eafd51547acb780d4aac5ebe7cc; no voters:
I20240502 20:45:58.779831 5825 raft_consensus.cc:2781] T ed6386b6777c4d50b29ff8cd49894eba P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:58.779911 5825 raft_consensus.cc:686] T ed6386b6777c4d50b29ff8cd49894eba P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Becoming Leader. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Running, Role: LEADER
I20240502 20:45:58.780054 5825 consensus_queue.cc:237] T ed6386b6777c4d50b29ff8cd49894eba P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:58.781749 4235 catalog_manager.cc:5484] T ed6386b6777c4d50b29ff8cd49894eba P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67). New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:58.906649 5915 consensus_queue.cc:1035] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:58.912544 5810 consensus_queue.cc:1035] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 20:45:58.950611 5931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:58.950852 5931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:58.954038 5931 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240502 20:45:59.016620 5864 consensus_queue.cc:1035] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:59.021018 5864 consensus_queue.cc:1035] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:59.039425 4768 raft_consensus.cc:1261] T ed6386b6777c4d50b29ff8cd49894eba P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:59.039898 5929 consensus_queue.cc:1035] T ed6386b6777c4d50b29ff8cd49894eba P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:59.045246 4506 raft_consensus.cc:1261] T ed6386b6777c4d50b29ff8cd49894eba P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:59.047499 5929 consensus_queue.cc:1035] T ed6386b6777c4d50b29ff8cd49894eba P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:59.077150 5960 mvcc.cc:205] Tried to move back new op lower bound from 7023340581021536256 to 7023340579964366848. Current Snapshot: MvccSnapshot[applied={T|T < 7023340581021536256}]
INSERT report
rows total: 2000
time total: 105.339 ms
time per row: 0.0526693 ms
I20240502 20:45:59.106565 4228 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41812:
name: "rebalance_test_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:45:59.112192 4749 tablet_service.cc:1467] Processing CreateTablet for tablet c0c95d50b8f64d829b7edbcd6f7c931e (DEFAULT_TABLE table=rebalance_test_table_10 [id=e54c8dd95147474b9f91de67bc9c075c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:59.112316 4357 tablet_service.cc:1467] Processing CreateTablet for tablet c0c95d50b8f64d829b7edbcd6f7c931e (DEFAULT_TABLE table=rebalance_test_table_10 [id=e54c8dd95147474b9f91de67bc9c075c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:59.112316 4488 tablet_service.cc:1467] Processing CreateTablet for tablet c0c95d50b8f64d829b7edbcd6f7c931e (DEFAULT_TABLE table=rebalance_test_table_10 [id=e54c8dd95147474b9f91de67bc9c075c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:59.112514 4749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0c95d50b8f64d829b7edbcd6f7c931e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:59.112608 4488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0c95d50b8f64d829b7edbcd6f7c931e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:59.112608 4357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0c95d50b8f64d829b7edbcd6f7c931e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:59.114415 5992 tablet_bootstrap.cc:492] T c0c95d50b8f64d829b7edbcd6f7c931e P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:45:59.114951 4619 tablet_service.cc:1467] Processing CreateTablet for tablet 789757df8e7d4284abcd7fcdf7538cf5 (DEFAULT_TABLE table=rebalance_test_table_10 [id=e54c8dd95147474b9f91de67bc9c075c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:59.115237 4619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 789757df8e7d4284abcd7fcdf7538cf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:59.115408 4355 tablet_service.cc:1467] Processing CreateTablet for tablet 9e34e8e17a7e413b97eb614a7df727d5 (DEFAULT_TABLE table=rebalance_test_table_10 [id=e54c8dd95147474b9f91de67bc9c075c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:59.115408 4488 tablet_service.cc:1467] Processing CreateTablet for tablet 9e34e8e17a7e413b97eb614a7df727d5 (DEFAULT_TABLE table=rebalance_test_table_10 [id=e54c8dd95147474b9f91de67bc9c075c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:59.115610 5992 tablet_bootstrap.cc:654] T c0c95d50b8f64d829b7edbcd6f7c931e P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:59.115640 4488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e34e8e17a7e413b97eb614a7df727d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:59.115640 4355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e34e8e17a7e413b97eb614a7df727d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:59.117573 4750 tablet_service.cc:1467] Processing CreateTablet for tablet 9e34e8e17a7e413b97eb614a7df727d5 (DEFAULT_TABLE table=rebalance_test_table_10 [id=e54c8dd95147474b9f91de67bc9c075c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:59.117835 4750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e34e8e17a7e413b97eb614a7df727d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:59.117877 4357 tablet_service.cc:1467] Processing CreateTablet for tablet 789757df8e7d4284abcd7fcdf7538cf5 (DEFAULT_TABLE table=rebalance_test_table_10 [id=e54c8dd95147474b9f91de67bc9c075c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:59.117962 5996 tablet_bootstrap.cc:492] T 789757df8e7d4284abcd7fcdf7538cf5 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:45:59.118100 4357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 789757df8e7d4284abcd7fcdf7538cf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:59.118283 5992 tablet_bootstrap.cc:492] T c0c95d50b8f64d829b7edbcd6f7c931e P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:45:59.118355 5992 ts_tablet_manager.cc:1367] T c0c95d50b8f64d829b7edbcd6f7c931e P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 20:45:59.118804 5992 raft_consensus.cc:348] T c0c95d50b8f64d829b7edbcd6f7c931e P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:59.118911 5992 raft_consensus.cc:374] T c0c95d50b8f64d829b7edbcd6f7c931e P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:59.118945 5996 tablet_bootstrap.cc:654] T 789757df8e7d4284abcd7fcdf7538cf5 P 923cfab926e7443f8ffd7b21ad0a9d91: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:59.118947 5992 raft_consensus.cc:729] T c0c95d50b8f64d829b7edbcd6f7c931e P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:45:59.119076 5992 consensus_queue.cc:260] T c0c95d50b8f64d829b7edbcd6f7c931e P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:59.119345 5992 ts_tablet_manager.cc:1398] T c0c95d50b8f64d829b7edbcd6f7c931e P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:59.120048 5996 tablet_bootstrap.cc:492] T 789757df8e7d4284abcd7fcdf7538cf5 P 923cfab926e7443f8ffd7b21ad0a9d91: No bootstrap required, opened a new log
I20240502 20:45:59.120127 5996 ts_tablet_manager.cc:1367] T 789757df8e7d4284abcd7fcdf7538cf5 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:59.120142 5993 tablet_bootstrap.cc:492] T c0c95d50b8f64d829b7edbcd6f7c931e P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:45:59.120585 5996 raft_consensus.cc:348] T 789757df8e7d4284abcd7fcdf7538cf5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:45:59.120704 5996 raft_consensus.cc:374] T 789757df8e7d4284abcd7fcdf7538cf5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:59.120738 5996 raft_consensus.cc:729] T 789757df8e7d4284abcd7fcdf7538cf5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: FOLLOWER
I20240502 20:45:59.120862 5996 consensus_queue.cc:260] T 789757df8e7d4284abcd7fcdf7538cf5 P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:45:59.121101 5993 tablet_bootstrap.cc:654] T c0c95d50b8f64d829b7edbcd6f7c931e P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:59.121223 5996 ts_tablet_manager.cc:1398] T 789757df8e7d4284abcd7fcdf7538cf5 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:59.121690 5992 tablet_bootstrap.cc:492] T 9e34e8e17a7e413b97eb614a7df727d5 P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:45:59.121745 5994 tablet_bootstrap.cc:492] T c0c95d50b8f64d829b7edbcd6f7c931e P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:45:59.122551 5994 tablet_bootstrap.cc:654] T c0c95d50b8f64d829b7edbcd6f7c931e P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:59.123006 5992 tablet_bootstrap.cc:654] T 9e34e8e17a7e413b97eb614a7df727d5 P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:59.123580 5994 tablet_bootstrap.cc:492] T c0c95d50b8f64d829b7edbcd6f7c931e P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:45:59.123665 5994 ts_tablet_manager.cc:1367] T c0c95d50b8f64d829b7edbcd6f7c931e P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:45:59.123808 4749 tablet_service.cc:1467] Processing CreateTablet for tablet 789757df8e7d4284abcd7fcdf7538cf5 (DEFAULT_TABLE table=rebalance_test_table_10 [id=e54c8dd95147474b9f91de67bc9c075c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:59.124037 4749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 789757df8e7d4284abcd7fcdf7538cf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:59.124080 5992 tablet_bootstrap.cc:492] T 9e34e8e17a7e413b97eb614a7df727d5 P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:45:59.124097 5994 raft_consensus.cc:348] T c0c95d50b8f64d829b7edbcd6f7c931e P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:59.124178 5992 ts_tablet_manager.cc:1367] T 9e34e8e17a7e413b97eb614a7df727d5 P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:45:59.124205 5994 raft_consensus.cc:374] T c0c95d50b8f64d829b7edbcd6f7c931e P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:59.124244 5994 raft_consensus.cc:729] T c0c95d50b8f64d829b7edbcd6f7c931e P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:45:59.124380 5994 consensus_queue.cc:260] T c0c95d50b8f64d829b7edbcd6f7c931e P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:59.124608 5992 raft_consensus.cc:348] T 9e34e8e17a7e413b97eb614a7df727d5 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:59.124673 5994 ts_tablet_manager.cc:1398] T c0c95d50b8f64d829b7edbcd6f7c931e P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:45:59.124701 5992 raft_consensus.cc:374] T 9e34e8e17a7e413b97eb614a7df727d5 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:59.125149 5993 tablet_bootstrap.cc:492] T c0c95d50b8f64d829b7edbcd6f7c931e P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:45:59.125241 5993 ts_tablet_manager.cc:1367] T c0c95d50b8f64d829b7edbcd6f7c931e P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240502 20:45:59.124737 5992 raft_consensus.cc:729] T 9e34e8e17a7e413b97eb614a7df727d5 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:45:59.125582 5992 consensus_queue.cc:260] T 9e34e8e17a7e413b97eb614a7df727d5 P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:59.125689 5993 raft_consensus.cc:348] T c0c95d50b8f64d829b7edbcd6f7c931e P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:59.125787 5993 raft_consensus.cc:374] T c0c95d50b8f64d829b7edbcd6f7c931e P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:59.125819 5993 raft_consensus.cc:729] T c0c95d50b8f64d829b7edbcd6f7c931e P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:45:59.125864 5992 ts_tablet_manager.cc:1398] T 9e34e8e17a7e413b97eb614a7df727d5 P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:45:59.125962 5993 consensus_queue.cc:260] T c0c95d50b8f64d829b7edbcd6f7c931e P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:59.126140 5992 tablet_bootstrap.cc:492] T 789757df8e7d4284abcd7fcdf7538cf5 P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:45:59.126300 5993 ts_tablet_manager.cc:1398] T c0c95d50b8f64d829b7edbcd6f7c931e P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:45:59.126438 5993 tablet_bootstrap.cc:492] T 9e34e8e17a7e413b97eb614a7df727d5 P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:45:59.126955 5992 tablet_bootstrap.cc:654] T 789757df8e7d4284abcd7fcdf7538cf5 P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:59.127995 5992 tablet_bootstrap.cc:492] T 789757df8e7d4284abcd7fcdf7538cf5 P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:45:59.128050 5994 tablet_bootstrap.cc:492] T 9e34e8e17a7e413b97eb614a7df727d5 P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:45:59.128073 5992 ts_tablet_manager.cc:1367] T 789757df8e7d4284abcd7fcdf7538cf5 P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:45:59.128464 5992 raft_consensus.cc:348] T 789757df8e7d4284abcd7fcdf7538cf5 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:45:59.128567 5992 raft_consensus.cc:374] T 789757df8e7d4284abcd7fcdf7538cf5 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:59.128600 5992 raft_consensus.cc:729] T 789757df8e7d4284abcd7fcdf7538cf5 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:45:59.128719 5992 consensus_queue.cc:260] T 789757df8e7d4284abcd7fcdf7538cf5 P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:45:59.129034 5992 ts_tablet_manager.cc:1398] T 789757df8e7d4284abcd7fcdf7538cf5 P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:59.129698 5994 tablet_bootstrap.cc:654] T 9e34e8e17a7e413b97eb614a7df727d5 P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:59.130640 5993 tablet_bootstrap.cc:654] T 9e34e8e17a7e413b97eb614a7df727d5 P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:59.131760 5993 tablet_bootstrap.cc:492] T 9e34e8e17a7e413b97eb614a7df727d5 P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:45:59.131856 5993 ts_tablet_manager.cc:1367] T 9e34e8e17a7e413b97eb614a7df727d5 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240502 20:45:59.132112 5994 tablet_bootstrap.cc:492] T 9e34e8e17a7e413b97eb614a7df727d5 P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:45:59.132200 5994 ts_tablet_manager.cc:1367] T 9e34e8e17a7e413b97eb614a7df727d5 P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 20:45:59.132333 5993 raft_consensus.cc:348] T 9e34e8e17a7e413b97eb614a7df727d5 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:59.132442 5993 raft_consensus.cc:374] T 9e34e8e17a7e413b97eb614a7df727d5 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:59.132489 5993 raft_consensus.cc:729] T 9e34e8e17a7e413b97eb614a7df727d5 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:45:59.132633 5993 consensus_queue.cc:260] T 9e34e8e17a7e413b97eb614a7df727d5 P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:59.132663 5994 raft_consensus.cc:348] T 9e34e8e17a7e413b97eb614a7df727d5 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:59.132761 5994 raft_consensus.cc:374] T 9e34e8e17a7e413b97eb614a7df727d5 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:59.132798 5994 raft_consensus.cc:729] T 9e34e8e17a7e413b97eb614a7df727d5 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:45:59.132941 5993 ts_tablet_manager.cc:1398] T 9e34e8e17a7e413b97eb614a7df727d5 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:45:59.132910 5994 consensus_queue.cc:260] T 9e34e8e17a7e413b97eb614a7df727d5 P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:59.133229 5994 ts_tablet_manager.cc:1398] T 9e34e8e17a7e413b97eb614a7df727d5 P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:59.133460 5994 tablet_bootstrap.cc:492] T 789757df8e7d4284abcd7fcdf7538cf5 P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:45:59.134342 5994 tablet_bootstrap.cc:654] T 789757df8e7d4284abcd7fcdf7538cf5 P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:59.136337 5994 tablet_bootstrap.cc:492] T 789757df8e7d4284abcd7fcdf7538cf5 P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:45:59.136427 5994 ts_tablet_manager.cc:1367] T 789757df8e7d4284abcd7fcdf7538cf5 P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 20:45:59.136932 5994 raft_consensus.cc:348] T 789757df8e7d4284abcd7fcdf7538cf5 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:45:59.137100 5994 raft_consensus.cc:374] T 789757df8e7d4284abcd7fcdf7538cf5 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:59.137147 5994 raft_consensus.cc:729] T 789757df8e7d4284abcd7fcdf7538cf5 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:45:59.137297 5994 consensus_queue.cc:260] T 789757df8e7d4284abcd7fcdf7538cf5 P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:45:59.137627 5994 ts_tablet_manager.cc:1398] T 789757df8e7d4284abcd7fcdf7538cf5 P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:59.162276 5380 raft_consensus.cc:482] T 9e34e8e17a7e413b97eb614a7df727d5 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:59.162403 5380 raft_consensus.cc:504] T 9e34e8e17a7e413b97eb614a7df727d5 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:59.162819 5380 leader_election.cc:288] T 9e34e8e17a7e413b97eb614a7df727d5 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827), c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097)
I20240502 20:45:59.163195 4506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e34e8e17a7e413b97eb614a7df727d5" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" is_pre_election: true
I20240502 20:45:59.163235 4375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e34e8e17a7e413b97eb614a7df727d5" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2" is_pre_election: true
I20240502 20:45:59.163343 4506 raft_consensus.cc:2454] T 9e34e8e17a7e413b97eb614a7df727d5 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 0.
I20240502 20:45:59.163393 4375 raft_consensus.cc:2454] T 9e34e8e17a7e413b97eb614a7df727d5 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 0.
I20240502 20:45:59.163657 4705 leader_election.cc:302] T 9e34e8e17a7e413b97eb614a7df727d5 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3d26eafd51547acb780d4aac5ebe7cc, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:45:59.163878 5380 raft_consensus.cc:2781] T 9e34e8e17a7e413b97eb614a7df727d5 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:59.163929 5380 raft_consensus.cc:482] T 9e34e8e17a7e413b97eb614a7df727d5 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:59.163982 5380 raft_consensus.cc:3037] T 9e34e8e17a7e413b97eb614a7df727d5 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:59.164847 5380 raft_consensus.cc:504] T 9e34e8e17a7e413b97eb614a7df727d5 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:59.165223 5380 leader_election.cc:288] T 9e34e8e17a7e413b97eb614a7df727d5 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 election: Requested vote from peers c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827), c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097)
I20240502 20:45:59.165597 4375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e34e8e17a7e413b97eb614a7df727d5" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2"
I20240502 20:45:59.165609 4506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e34e8e17a7e413b97eb614a7df727d5" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc"
I20240502 20:45:59.165721 4375 raft_consensus.cc:3037] T 9e34e8e17a7e413b97eb614a7df727d5 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:59.165727 4506 raft_consensus.cc:3037] T 9e34e8e17a7e413b97eb614a7df727d5 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:59.166532 4506 raft_consensus.cc:2454] T 9e34e8e17a7e413b97eb614a7df727d5 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 1.
I20240502 20:45:59.166538 4375 raft_consensus.cc:2454] T 9e34e8e17a7e413b97eb614a7df727d5 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 1.
I20240502 20:45:59.166851 4706 leader_election.cc:302] T 9e34e8e17a7e413b97eb614a7df727d5 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:45:59.167009 5380 raft_consensus.cc:2781] T 9e34e8e17a7e413b97eb614a7df727d5 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:59.167102 5380 raft_consensus.cc:686] T 9e34e8e17a7e413b97eb614a7df727d5 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Becoming Leader. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Running, Role: LEADER
I20240502 20:45:59.167223 5380 consensus_queue.cc:237] T 9e34e8e17a7e413b97eb614a7df727d5 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:59.169184 4228 catalog_manager.cc:5484] T 9e34e8e17a7e413b97eb614a7df727d5 P ed47c023452a497192f35d2f617bf9bd reported cstate change: term changed from 0 to 1, leader changed from <none> to ed47c023452a497192f35d2f617bf9bd (127.30.32.68). New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:59.258481 5864 raft_consensus.cc:482] T c0c95d50b8f64d829b7edbcd6f7c931e P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:59.258620 5864 raft_consensus.cc:504] T c0c95d50b8f64d829b7edbcd6f7c931e P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:59.259037 5864 leader_election.cc:288] T c0c95d50b8f64d829b7edbcd6f7c931e P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193), c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:45:59.259444 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0c95d50b8f64d829b7edbcd6f7c931e" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd" is_pre_election: true
I20240502 20:45:59.259492 4375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0c95d50b8f64d829b7edbcd6f7c931e" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2" is_pre_election: true
I20240502 20:45:59.259569 4768 raft_consensus.cc:2454] T c0c95d50b8f64d829b7edbcd6f7c931e P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 0.
I20240502 20:45:59.259660 4375 raft_consensus.cc:2454] T c0c95d50b8f64d829b7edbcd6f7c931e P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 0.
I20240502 20:45:59.259918 4444 leader_election.cc:302] T c0c95d50b8f64d829b7edbcd6f7c931e P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3d26eafd51547acb780d4aac5ebe7cc, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:45:59.260082 5864 raft_consensus.cc:2781] T c0c95d50b8f64d829b7edbcd6f7c931e P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:59.260140 5864 raft_consensus.cc:482] T c0c95d50b8f64d829b7edbcd6f7c931e P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:59.260177 5864 raft_consensus.cc:3037] T c0c95d50b8f64d829b7edbcd6f7c931e P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:59.260972 5864 raft_consensus.cc:504] T c0c95d50b8f64d829b7edbcd6f7c931e P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:59.261323 5864 leader_election.cc:288] T c0c95d50b8f64d829b7edbcd6f7c931e P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 election: Requested vote from peers ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193), c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:45:59.261624 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0c95d50b8f64d829b7edbcd6f7c931e" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd"
I20240502 20:45:59.261744 4768 raft_consensus.cc:3037] T c0c95d50b8f64d829b7edbcd6f7c931e P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:59.261726 4375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0c95d50b8f64d829b7edbcd6f7c931e" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2"
I20240502 20:45:59.261840 4375 raft_consensus.cc:3037] T c0c95d50b8f64d829b7edbcd6f7c931e P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:59.262498 4768 raft_consensus.cc:2454] T c0c95d50b8f64d829b7edbcd6f7c931e P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 1.
I20240502 20:45:59.262624 4375 raft_consensus.cc:2454] T c0c95d50b8f64d829b7edbcd6f7c931e P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 1.
I20240502 20:45:59.262784 4444 leader_election.cc:302] T c0c95d50b8f64d829b7edbcd6f7c931e P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3d26eafd51547acb780d4aac5ebe7cc, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:45:59.263059 5864 raft_consensus.cc:2781] T c0c95d50b8f64d829b7edbcd6f7c931e P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:59.263156 5864 raft_consensus.cc:686] T c0c95d50b8f64d829b7edbcd6f7c931e P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Becoming Leader. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Running, Role: LEADER
I20240502 20:45:59.263296 5864 consensus_queue.cc:237] T c0c95d50b8f64d829b7edbcd6f7c931e P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:59.265412 4228 catalog_manager.cc:5484] T c0c95d50b8f64d829b7edbcd6f7c931e P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: term changed from 0 to 1, leader changed from <none> to c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66). New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } }
I20240502 20:45:59.278860 5984 raft_consensus.cc:482] T 789757df8e7d4284abcd7fcdf7538cf5 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:59.278980 5984 raft_consensus.cc:504] T 789757df8e7d4284abcd7fcdf7538cf5 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:45:59.279395 5984 leader_election.cc:288] T 789757df8e7d4284abcd7fcdf7538cf5 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193), 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:45:59.279690 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "789757df8e7d4284abcd7fcdf7538cf5" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd" is_pre_election: true
I20240502 20:45:59.279731 4639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "789757df8e7d4284abcd7fcdf7538cf5" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" is_pre_election: true
I20240502 20:45:59.279835 4768 raft_consensus.cc:2454] T 789757df8e7d4284abcd7fcdf7538cf5 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 0.
I20240502 20:45:59.279860 4639 raft_consensus.cc:2454] T 789757df8e7d4284abcd7fcdf7538cf5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 0.
I20240502 20:45:59.280098 4313 leader_election.cc:302] T 789757df8e7d4284abcd7fcdf7538cf5 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:45:59.280257 5984 raft_consensus.cc:2781] T 789757df8e7d4284abcd7fcdf7538cf5 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:59.280325 5984 raft_consensus.cc:482] T 789757df8e7d4284abcd7fcdf7538cf5 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:59.280372 5984 raft_consensus.cc:3037] T 789757df8e7d4284abcd7fcdf7538cf5 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:59.281275 5984 raft_consensus.cc:504] T 789757df8e7d4284abcd7fcdf7538cf5 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:45:59.281566 5984 leader_election.cc:288] T 789757df8e7d4284abcd7fcdf7538cf5 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 election: Requested vote from peers ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193), 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:45:59.281857 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "789757df8e7d4284abcd7fcdf7538cf5" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd"
I20240502 20:45:59.281893 4639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "789757df8e7d4284abcd7fcdf7538cf5" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91"
I20240502 20:45:59.281961 4768 raft_consensus.cc:3037] T 789757df8e7d4284abcd7fcdf7538cf5 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:59.282002 4639 raft_consensus.cc:3037] T 789757df8e7d4284abcd7fcdf7538cf5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:59.282655 4768 raft_consensus.cc:2454] T 789757df8e7d4284abcd7fcdf7538cf5 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 1.
I20240502 20:45:59.282805 4639 raft_consensus.cc:2454] T 789757df8e7d4284abcd7fcdf7538cf5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 1.
I20240502 20:45:59.282931 4313 leader_election.cc:302] T 789757df8e7d4284abcd7fcdf7538cf5 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:45:59.283104 5984 raft_consensus.cc:2781] T 789757df8e7d4284abcd7fcdf7538cf5 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:59.283191 5984 raft_consensus.cc:686] T 789757df8e7d4284abcd7fcdf7538cf5 P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEADER]: Becoming Leader. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Running, Role: LEADER
I20240502 20:45:59.283331 5984 consensus_queue.cc:237] T 789757df8e7d4284abcd7fcdf7538cf5 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:45:59.285773 4228 catalog_manager.cc:5484] T 789757df8e7d4284abcd7fcdf7538cf5 P c92e6e46782849479c8a0c6d82276fb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65). New cstate: current_term: 1 leader_uuid: "c92e6e46782849479c8a0c6d82276fb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: UNKNOWN } } }
W20240502 20:45:59.422233 6008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:59.422461 6008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:59.425504 6008 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_10'
I20240502 20:45:59.523692 4768 raft_consensus.cc:1261] T 789757df8e7d4284abcd7fcdf7538cf5 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:59.523996 4639 raft_consensus.cc:1261] T 789757df8e7d4284abcd7fcdf7538cf5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:59.524060 4770 raft_consensus.cc:1261] T c0c95d50b8f64d829b7edbcd6f7c931e P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:45:59.524243 4375 raft_consensus.cc:1261] T c0c95d50b8f64d829b7edbcd6f7c931e P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:45:59.524761 5864 consensus_queue.cc:1035] T c0c95d50b8f64d829b7edbcd6f7c931e P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:59.524874 5984 consensus_queue.cc:1035] T 789757df8e7d4284abcd7fcdf7538cf5 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:59.524926 4506 raft_consensus.cc:1261] T 9e34e8e17a7e413b97eb614a7df727d5 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:59.525156 5864 consensus_queue.cc:1035] T c0c95d50b8f64d829b7edbcd6f7c931e P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:59.525424 5380 consensus_queue.cc:1035] T 9e34e8e17a7e413b97eb614a7df727d5 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:59.525455 5915 consensus_queue.cc:1035] T 789757df8e7d4284abcd7fcdf7538cf5 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:59.525873 4375 raft_consensus.cc:1261] T 9e34e8e17a7e413b97eb614a7df727d5 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:59.528066 5380 consensus_queue.cc:1035] T 9e34e8e17a7e413b97eb614a7df727d5 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:59.544493 6042 mvcc.cc:205] Tried to move back new op lower bound from 7023340583003906048 to 7023340582025506816. Current Snapshot: MvccSnapshot[applied={T|T < 7023340583003906048}]
INSERT report
rows total: 2000
time total: 114.917 ms
time per row: 0.0574585 ms
I20240502 20:45:59.594812 4228 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41812:
name: "rebalance_test_table_11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:45:59.600257 4619 tablet_service.cc:1467] Processing CreateTablet for tablet 61f8c8d2925d46bdad8e903b48c3046e (DEFAULT_TABLE table=rebalance_test_table_11 [id=105c28a926194705956796da40cf3dab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:59.600358 4357 tablet_service.cc:1467] Processing CreateTablet for tablet 61f8c8d2925d46bdad8e903b48c3046e (DEFAULT_TABLE table=rebalance_test_table_11 [id=105c28a926194705956796da40cf3dab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:59.600593 4619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61f8c8d2925d46bdad8e903b48c3046e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:59.600612 4357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61f8c8d2925d46bdad8e903b48c3046e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:59.601869 4486 tablet_service.cc:1467] Processing CreateTablet for tablet 8565ebbd21dd4f708a0254d1ed2d6670 (DEFAULT_TABLE table=rebalance_test_table_11 [id=105c28a926194705956796da40cf3dab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:59.601938 4617 tablet_service.cc:1467] Processing CreateTablet for tablet 8565ebbd21dd4f708a0254d1ed2d6670 (DEFAULT_TABLE table=rebalance_test_table_11 [id=105c28a926194705956796da40cf3dab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:59.602155 4486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8565ebbd21dd4f708a0254d1ed2d6670. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:59.602465 4617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8565ebbd21dd4f708a0254d1ed2d6670. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:59.603041 4488 tablet_service.cc:1467] Processing CreateTablet for tablet 61f8c8d2925d46bdad8e903b48c3046e (DEFAULT_TABLE table=rebalance_test_table_11 [id=105c28a926194705956796da40cf3dab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:59.603287 4488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61f8c8d2925d46bdad8e903b48c3046e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:59.604435 4618 tablet_service.cc:1467] Processing CreateTablet for tablet eebddad3efe748f78372901ecad5b407 (DEFAULT_TABLE table=rebalance_test_table_11 [id=105c28a926194705956796da40cf3dab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:59.604672 4618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eebddad3efe748f78372901ecad5b407. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:59.605224 5993 tablet_bootstrap.cc:492] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:45:59.605224 5996 tablet_bootstrap.cc:492] T 61f8c8d2925d46bdad8e903b48c3046e P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:45:59.606063 5993 tablet_bootstrap.cc:654] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:59.606063 5996 tablet_bootstrap.cc:654] T 61f8c8d2925d46bdad8e903b48c3046e P 923cfab926e7443f8ffd7b21ad0a9d91: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:59.607563 5996 tablet_bootstrap.cc:492] T 61f8c8d2925d46bdad8e903b48c3046e P 923cfab926e7443f8ffd7b21ad0a9d91: No bootstrap required, opened a new log
I20240502 20:45:59.607659 5996 ts_tablet_manager.cc:1367] T 61f8c8d2925d46bdad8e903b48c3046e P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:45:59.608106 5996 raft_consensus.cc:348] T 61f8c8d2925d46bdad8e903b48c3046e P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:59.608222 5996 raft_consensus.cc:374] T 61f8c8d2925d46bdad8e903b48c3046e P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:59.608273 5996 raft_consensus.cc:729] T 61f8c8d2925d46bdad8e903b48c3046e P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: FOLLOWER
I20240502 20:45:59.608397 5996 consensus_queue.cc:260] T 61f8c8d2925d46bdad8e903b48c3046e P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:59.608682 5996 ts_tablet_manager.cc:1398] T 61f8c8d2925d46bdad8e903b48c3046e P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:59.608795 4355 tablet_service.cc:1467] Processing CreateTablet for tablet eebddad3efe748f78372901ecad5b407 (DEFAULT_TABLE table=rebalance_test_table_11 [id=105c28a926194705956796da40cf3dab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:59.608810 4749 tablet_service.cc:1467] Processing CreateTablet for tablet 8565ebbd21dd4f708a0254d1ed2d6670 (DEFAULT_TABLE table=rebalance_test_table_11 [id=105c28a926194705956796da40cf3dab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:59.608739 4750 tablet_service.cc:1467] Processing CreateTablet for tablet eebddad3efe748f78372901ecad5b407 (DEFAULT_TABLE table=rebalance_test_table_11 [id=105c28a926194705956796da40cf3dab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:59.609292 4355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eebddad3efe748f78372901ecad5b407. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:59.609697 5996 tablet_bootstrap.cc:492] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:45:59.609664 4749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8565ebbd21dd4f708a0254d1ed2d6670. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:59.610534 5996 tablet_bootstrap.cc:654] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:59.611505 5994 tablet_bootstrap.cc:492] T eebddad3efe748f78372901ecad5b407 P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:45:59.611982 5992 tablet_bootstrap.cc:492] T 8565ebbd21dd4f708a0254d1ed2d6670 P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:45:59.612334 5994 tablet_bootstrap.cc:654] T eebddad3efe748f78372901ecad5b407 P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:59.612879 5992 tablet_bootstrap.cc:654] T 8565ebbd21dd4f708a0254d1ed2d6670 P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:59.613536 4750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eebddad3efe748f78372901ecad5b407. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:59.613658 5994 tablet_bootstrap.cc:492] T eebddad3efe748f78372901ecad5b407 P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:45:59.613740 5994 ts_tablet_manager.cc:1367] T eebddad3efe748f78372901ecad5b407 P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:45:59.613991 5992 tablet_bootstrap.cc:492] T 8565ebbd21dd4f708a0254d1ed2d6670 P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:45:59.614061 5992 ts_tablet_manager.cc:1367] T 8565ebbd21dd4f708a0254d1ed2d6670 P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:59.614146 5994 raft_consensus.cc:348] T eebddad3efe748f78372901ecad5b407 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:59.614245 5994 raft_consensus.cc:374] T eebddad3efe748f78372901ecad5b407 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:59.614274 5994 raft_consensus.cc:729] T eebddad3efe748f78372901ecad5b407 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:45:59.614403 5994 consensus_queue.cc:260] T eebddad3efe748f78372901ecad5b407 P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:59.614514 5992 raft_consensus.cc:348] T 8565ebbd21dd4f708a0254d1ed2d6670 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:59.614615 5992 raft_consensus.cc:374] T 8565ebbd21dd4f708a0254d1ed2d6670 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:59.614651 5992 raft_consensus.cc:729] T 8565ebbd21dd4f708a0254d1ed2d6670 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:45:59.614714 5994 ts_tablet_manager.cc:1398] T eebddad3efe748f78372901ecad5b407 P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:59.614773 5992 consensus_queue.cc:260] T 8565ebbd21dd4f708a0254d1ed2d6670 P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:59.615020 5992 ts_tablet_manager.cc:1398] T 8565ebbd21dd4f708a0254d1ed2d6670 P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:59.615971 5992 tablet_bootstrap.cc:492] T eebddad3efe748f78372901ecad5b407 P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:45:59.616901 5992 tablet_bootstrap.cc:654] T eebddad3efe748f78372901ecad5b407 P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:59.617501 5994 tablet_bootstrap.cc:492] T 61f8c8d2925d46bdad8e903b48c3046e P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:45:59.617647 5993 tablet_bootstrap.cc:492] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:45:59.617779 5993 ts_tablet_manager.cc:1367] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.002s
I20240502 20:45:59.617791 5996 tablet_bootstrap.cc:492] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91: No bootstrap required, opened a new log
I20240502 20:45:59.617867 5996 ts_tablet_manager.cc:1367] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 20:45:59.618288 5996 raft_consensus.cc:348] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:59.618266 5993 raft_consensus.cc:348] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:59.618391 5996 raft_consensus.cc:374] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:59.618391 5993 raft_consensus.cc:374] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:59.618427 5993 raft_consensus.cc:729] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:45:59.618427 5996 raft_consensus.cc:729] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: FOLLOWER
I20240502 20:45:59.618533 5994 tablet_bootstrap.cc:654] T 61f8c8d2925d46bdad8e903b48c3046e P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:59.618546 5996 consensus_queue.cc:260] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:59.618546 5993 consensus_queue.cc:260] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:59.618808 5996 ts_tablet_manager.cc:1398] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:45:59.618853 5993 ts_tablet_manager.cc:1398] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:45:59.618983 5996 tablet_bootstrap.cc:492] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:45:59.619011 5993 tablet_bootstrap.cc:492] T 61f8c8d2925d46bdad8e903b48c3046e P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:45:59.619861 5996 tablet_bootstrap.cc:654] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:59.620038 5993 tablet_bootstrap.cc:654] T 61f8c8d2925d46bdad8e903b48c3046e P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:59.620362 5992 tablet_bootstrap.cc:492] T eebddad3efe748f78372901ecad5b407 P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:45:59.620441 5992 ts_tablet_manager.cc:1367] T eebddad3efe748f78372901ecad5b407 P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 20:45:59.620936 5992 raft_consensus.cc:348] T eebddad3efe748f78372901ecad5b407 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:59.621066 5992 raft_consensus.cc:374] T eebddad3efe748f78372901ecad5b407 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:59.621114 5992 raft_consensus.cc:729] T eebddad3efe748f78372901ecad5b407 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:45:59.621240 5992 consensus_queue.cc:260] T eebddad3efe748f78372901ecad5b407 P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:59.621555 5992 ts_tablet_manager.cc:1398] T eebddad3efe748f78372901ecad5b407 P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:59.622287 5994 tablet_bootstrap.cc:492] T 61f8c8d2925d46bdad8e903b48c3046e P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:45:59.622370 5994 ts_tablet_manager.cc:1367] T 61f8c8d2925d46bdad8e903b48c3046e P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 20:45:59.622824 5994 raft_consensus.cc:348] T 61f8c8d2925d46bdad8e903b48c3046e P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:59.622931 5994 raft_consensus.cc:374] T 61f8c8d2925d46bdad8e903b48c3046e P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:59.622960 5994 raft_consensus.cc:729] T 61f8c8d2925d46bdad8e903b48c3046e P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:45:59.623071 5994 consensus_queue.cc:260] T 61f8c8d2925d46bdad8e903b48c3046e P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:59.623291 5996 tablet_bootstrap.cc:492] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91: No bootstrap required, opened a new log
I20240502 20:45:59.623366 5994 ts_tablet_manager.cc:1398] T 61f8c8d2925d46bdad8e903b48c3046e P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:59.623396 5996 ts_tablet_manager.cc:1367] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 20:45:59.623800 5996 raft_consensus.cc:348] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:59.623944 5996 raft_consensus.cc:374] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:59.623980 5996 raft_consensus.cc:729] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: FOLLOWER
I20240502 20:45:59.624121 5996 consensus_queue.cc:260] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:59.624462 5996 ts_tablet_manager.cc:1398] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:45:59.625069 5993 tablet_bootstrap.cc:492] T 61f8c8d2925d46bdad8e903b48c3046e P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:45:59.625155 5993 ts_tablet_manager.cc:1367] T 61f8c8d2925d46bdad8e903b48c3046e P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240502 20:45:59.625576 5993 raft_consensus.cc:348] T 61f8c8d2925d46bdad8e903b48c3046e P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:59.625659 5993 raft_consensus.cc:374] T 61f8c8d2925d46bdad8e903b48c3046e P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:59.625679 5993 raft_consensus.cc:729] T 61f8c8d2925d46bdad8e903b48c3046e P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:45:59.625783 5993 consensus_queue.cc:260] T 61f8c8d2925d46bdad8e903b48c3046e P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:59.626062 5993 ts_tablet_manager.cc:1398] T 61f8c8d2925d46bdad8e903b48c3046e P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:45:59.723537 5825 raft_consensus.cc:482] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:59.723701 5825 raft_consensus.cc:504] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:59.724136 5825 leader_election.cc:288] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193), c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097)
I20240502 20:45:59.724546 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8565ebbd21dd4f708a0254d1ed2d6670" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd" is_pre_election: true
I20240502 20:45:59.724586 4506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8565ebbd21dd4f708a0254d1ed2d6670" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" is_pre_election: true
I20240502 20:45:59.724707 4768 raft_consensus.cc:2454] T 8565ebbd21dd4f708a0254d1ed2d6670 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 0.
I20240502 20:45:59.724706 4506 raft_consensus.cc:2454] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 0.
I20240502 20:45:59.725034 4574 leader_election.cc:302] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, c3d26eafd51547acb780d4aac5ebe7cc; no voters:
I20240502 20:45:59.725229 5825 raft_consensus.cc:2781] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:59.725317 5825 raft_consensus.cc:482] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:59.725370 5825 raft_consensus.cc:3037] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:59.726203 5825 raft_consensus.cc:504] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:59.726540 5825 leader_election.cc:288] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 election: Requested vote from peers ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193), c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097)
I20240502 20:45:59.726891 4506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8565ebbd21dd4f708a0254d1ed2d6670" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc"
I20240502 20:45:59.726882 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8565ebbd21dd4f708a0254d1ed2d6670" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd"
I20240502 20:45:59.727005 4768 raft_consensus.cc:3037] T 8565ebbd21dd4f708a0254d1ed2d6670 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:59.727012 4506 raft_consensus.cc:3037] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:59.727782 4506 raft_consensus.cc:2454] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 1.
I20240502 20:45:59.727789 4768 raft_consensus.cc:2454] T 8565ebbd21dd4f708a0254d1ed2d6670 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 1.
I20240502 20:45:59.728078 4575 leader_election.cc:302] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:45:59.728256 5825 raft_consensus.cc:2781] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:59.728338 5825 raft_consensus.cc:686] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Becoming Leader. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Running, Role: LEADER
I20240502 20:45:59.728485 5825 consensus_queue.cc:237] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:59.730373 4235 catalog_manager.cc:5484] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67). New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:59.769165 6036 raft_consensus.cc:482] T 61f8c8d2925d46bdad8e903b48c3046e P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:59.769356 6036 raft_consensus.cc:504] T 61f8c8d2925d46bdad8e903b48c3046e P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:59.769924 6036 leader_election.cc:288] T 61f8c8d2925d46bdad8e903b48c3046e P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141), c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:45:59.770373 4639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f8c8d2925d46bdad8e903b48c3046e" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" is_pre_election: true
I20240502 20:45:59.770443 4376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f8c8d2925d46bdad8e903b48c3046e" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2" is_pre_election: true
I20240502 20:45:59.770542 4639 raft_consensus.cc:2454] T 61f8c8d2925d46bdad8e903b48c3046e P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 0.
I20240502 20:45:59.770574 4376 raft_consensus.cc:2454] T 61f8c8d2925d46bdad8e903b48c3046e P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 0.
I20240502 20:45:59.770906 4441 leader_election.cc:302] T 61f8c8d2925d46bdad8e903b48c3046e P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, c3d26eafd51547acb780d4aac5ebe7cc; no voters:
I20240502 20:45:59.771093 6036 raft_consensus.cc:2781] T 61f8c8d2925d46bdad8e903b48c3046e P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:59.771169 6036 raft_consensus.cc:482] T 61f8c8d2925d46bdad8e903b48c3046e P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:59.771198 6036 raft_consensus.cc:3037] T 61f8c8d2925d46bdad8e903b48c3046e P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:59.772132 6036 raft_consensus.cc:504] T 61f8c8d2925d46bdad8e903b48c3046e P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:59.772483 6036 leader_election.cc:288] T 61f8c8d2925d46bdad8e903b48c3046e P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 election: Requested vote from peers 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141), c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:45:59.772876 4639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f8c8d2925d46bdad8e903b48c3046e" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91"
I20240502 20:45:59.772920 4376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f8c8d2925d46bdad8e903b48c3046e" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2"
I20240502 20:45:59.773034 4639 raft_consensus.cc:3037] T 61f8c8d2925d46bdad8e903b48c3046e P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:59.773085 4376 raft_consensus.cc:3037] T 61f8c8d2925d46bdad8e903b48c3046e P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:59.774114 4639 raft_consensus.cc:2454] T 61f8c8d2925d46bdad8e903b48c3046e P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 1.
I20240502 20:45:59.774191 4376 raft_consensus.cc:2454] T 61f8c8d2925d46bdad8e903b48c3046e P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 1.
I20240502 20:45:59.774434 4441 leader_election.cc:302] T 61f8c8d2925d46bdad8e903b48c3046e P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, c3d26eafd51547acb780d4aac5ebe7cc; no voters:
I20240502 20:45:59.774655 6036 raft_consensus.cc:2781] T 61f8c8d2925d46bdad8e903b48c3046e P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:59.774755 6036 raft_consensus.cc:686] T 61f8c8d2925d46bdad8e903b48c3046e P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Becoming Leader. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Running, Role: LEADER
I20240502 20:45:59.774904 6036 consensus_queue.cc:237] T 61f8c8d2925d46bdad8e903b48c3046e P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:45:59.776852 4235 catalog_manager.cc:5484] T 61f8c8d2925d46bdad8e903b48c3046e P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: term changed from 0 to 1, leader changed from <none> to c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66). New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } }
I20240502 20:45:59.891242 5929 raft_consensus.cc:482] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:59.891381 5929 raft_consensus.cc:504] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:59.891737 5929 leader_election.cc:288] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827), ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:45:59.892171 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eebddad3efe748f78372901ecad5b407" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd" is_pre_election: true
I20240502 20:45:59.892171 4376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eebddad3efe748f78372901ecad5b407" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2" is_pre_election: true
I20240502 20:45:59.892329 4376 raft_consensus.cc:2454] T eebddad3efe748f78372901ecad5b407 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 0.
I20240502 20:45:59.892329 4768 raft_consensus.cc:2454] T eebddad3efe748f78372901ecad5b407 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 0.
I20240502 20:45:59.892632 4575 leader_election.cc:302] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:45:59.892792 5929 raft_consensus.cc:2781] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:59.892855 5929 raft_consensus.cc:482] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:59.892921 5929 raft_consensus.cc:3037] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:59.893817 5929 raft_consensus.cc:504] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:59.894096 5929 leader_election.cc:288] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 election: Requested vote from peers c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827), ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:45:59.894433 4376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eebddad3efe748f78372901ecad5b407" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2"
I20240502 20:45:59.894531 4376 raft_consensus.cc:3037] T eebddad3efe748f78372901ecad5b407 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:59.894487 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eebddad3efe748f78372901ecad5b407" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd"
I20240502 20:45:59.894615 4768 raft_consensus.cc:3037] T eebddad3efe748f78372901ecad5b407 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:59.895296 4376 raft_consensus.cc:2454] T eebddad3efe748f78372901ecad5b407 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 1.
I20240502 20:45:59.895385 4768 raft_consensus.cc:2454] T eebddad3efe748f78372901ecad5b407 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 1.
I20240502 20:45:59.895572 4575 leader_election.cc:302] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:45:59.895761 5929 raft_consensus.cc:2781] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:59.895848 5929 raft_consensus.cc:686] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Becoming Leader. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Running, Role: LEADER
I20240502 20:45:59.895958 5929 consensus_queue.cc:237] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:45:59.897753 4235 catalog_manager.cc:5484] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67). New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: UNKNOWN } } }
W20240502 20:46:00.027491 6062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:00.027802 6062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:00.032043 6062 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_11'
I20240502 20:46:00.115250 4768 raft_consensus.cc:1261] T 8565ebbd21dd4f708a0254d1ed2d6670 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:00.115643 4639 raft_consensus.cc:1261] T 61f8c8d2925d46bdad8e903b48c3046e P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:00.115732 4770 raft_consensus.cc:1261] T eebddad3efe748f78372901ecad5b407 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:00.116077 4376 raft_consensus.cc:1261] T 61f8c8d2925d46bdad8e903b48c3046e P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:00.116154 6037 consensus_queue.cc:1035] T 61f8c8d2925d46bdad8e903b48c3046e P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:00.116534 4375 raft_consensus.cc:1261] T eebddad3efe748f78372901ecad5b407 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:00.116847 4506 raft_consensus.cc:1261] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:00.117601 6037 consensus_queue.cc:1035] T 61f8c8d2925d46bdad8e903b48c3046e P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:00.117830 5825 consensus_queue.cc:1035] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:00.118137 5825 consensus_queue.cc:1035] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:00.118456 5825 consensus_queue.cc:1035] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:00.121726 5929 consensus_queue.cc:1035] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:00.122051 6093 mvcc.cc:205] Tried to move back new op lower bound from 7023340585422479360 to 7023340584039387136. Current Snapshot: MvccSnapshot[applied={T|T < 7023340585422479360}]
I20240502 20:46:00.122957 6094 mvcc.cc:205] Tried to move back new op lower bound from 7023340585422479360 to 7023340584039387136. Current Snapshot: MvccSnapshot[applied={T|T < 7023340585422479360}]
INSERT report
rows total: 2000
time total: 106.265 ms
time per row: 0.0531327 ms
I20240502 20:46:00.185936 4231 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41812:
name: "rebalance_test_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:46:00.191509 4488 tablet_service.cc:1467] Processing CreateTablet for tablet be40862863f44123916c80a1ae5b24a9 (DEFAULT_TABLE table=rebalance_test_table_12 [id=dab4c58041724066a7c74bff06307285]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:00.191844 4488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be40862863f44123916c80a1ae5b24a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:00.191896 4486 tablet_service.cc:1467] Processing CreateTablet for tablet 6faa71d4bd2143b0a72cd86a7e1e7e12 (DEFAULT_TABLE table=rebalance_test_table_12 [id=dab4c58041724066a7c74bff06307285]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:00.192150 4486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6faa71d4bd2143b0a72cd86a7e1e7e12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:00.192546 4750 tablet_service.cc:1467] Processing CreateTablet for tablet 6faa71d4bd2143b0a72cd86a7e1e7e12 (DEFAULT_TABLE table=rebalance_test_table_12 [id=dab4c58041724066a7c74bff06307285]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:00.192817 4750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6faa71d4bd2143b0a72cd86a7e1e7e12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:00.194006 6123 tablet_bootstrap.cc:492] T be40862863f44123916c80a1ae5b24a9 P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:46:00.194236 4357 tablet_service.cc:1467] Processing CreateTablet for tablet be40862863f44123916c80a1ae5b24a9 (DEFAULT_TABLE table=rebalance_test_table_12 [id=dab4c58041724066a7c74bff06307285]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:00.194505 4357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be40862863f44123916c80a1ae5b24a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:00.194895 6124 tablet_bootstrap.cc:492] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:46:00.194908 6123 tablet_bootstrap.cc:654] T be40862863f44123916c80a1ae5b24a9 P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:00.195611 4355 tablet_service.cc:1467] Processing CreateTablet for tablet 6faa71d4bd2143b0a72cd86a7e1e7e12 (DEFAULT_TABLE table=rebalance_test_table_12 [id=dab4c58041724066a7c74bff06307285]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:00.195849 4355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6faa71d4bd2143b0a72cd86a7e1e7e12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:00.195853 6124 tablet_bootstrap.cc:654] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:00.195904 4750 tablet_service.cc:1467] Processing CreateTablet for tablet f352efa74d3c4da3907efdc37237211f (DEFAULT_TABLE table=rebalance_test_table_12 [id=dab4c58041724066a7c74bff06307285]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:00.196148 4750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f352efa74d3c4da3907efdc37237211f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:00.197028 4487 tablet_service.cc:1467] Processing CreateTablet for tablet f352efa74d3c4da3907efdc37237211f (DEFAULT_TABLE table=rebalance_test_table_12 [id=dab4c58041724066a7c74bff06307285]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:00.197250 4487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f352efa74d3c4da3907efdc37237211f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:00.197247 6127 tablet_bootstrap.cc:492] T be40862863f44123916c80a1ae5b24a9 P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:46:00.198120 6127 tablet_bootstrap.cc:654] T be40862863f44123916c80a1ae5b24a9 P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:00.198316 4617 tablet_service.cc:1467] Processing CreateTablet for tablet f352efa74d3c4da3907efdc37237211f (DEFAULT_TABLE table=rebalance_test_table_12 [id=dab4c58041724066a7c74bff06307285]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:00.198590 4617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f352efa74d3c4da3907efdc37237211f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:00.199180 6127 tablet_bootstrap.cc:492] T be40862863f44123916c80a1ae5b24a9 P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:46:00.199244 6127 ts_tablet_manager.cc:1367] T be40862863f44123916c80a1ae5b24a9 P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:00.199683 6127 raft_consensus.cc:348] T be40862863f44123916c80a1ae5b24a9 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:00.199699 4618 tablet_service.cc:1467] Processing CreateTablet for tablet be40862863f44123916c80a1ae5b24a9 (DEFAULT_TABLE table=rebalance_test_table_12 [id=dab4c58041724066a7c74bff06307285]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:00.199777 6127 raft_consensus.cc:374] T be40862863f44123916c80a1ae5b24a9 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:00.199800 6127 raft_consensus.cc:729] T be40862863f44123916c80a1ae5b24a9 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:46:00.199923 4618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be40862863f44123916c80a1ae5b24a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:00.199914 6127 consensus_queue.cc:260] T be40862863f44123916c80a1ae5b24a9 P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:00.200181 6127 ts_tablet_manager.cc:1398] T be40862863f44123916c80a1ae5b24a9 P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:00.200299 6127 tablet_bootstrap.cc:492] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:46:00.200911 6129 tablet_bootstrap.cc:492] T f352efa74d3c4da3907efdc37237211f P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:46:00.201097 6127 tablet_bootstrap.cc:654] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:00.202004 6123 tablet_bootstrap.cc:492] T be40862863f44123916c80a1ae5b24a9 P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:46:00.202046 6129 tablet_bootstrap.cc:654] T f352efa74d3c4da3907efdc37237211f P 923cfab926e7443f8ffd7b21ad0a9d91: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:00.202092 6123 ts_tablet_manager.cc:1367] T be40862863f44123916c80a1ae5b24a9 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240502 20:46:00.202165 6127 tablet_bootstrap.cc:492] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:46:00.202241 6127 ts_tablet_manager.cc:1367] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 20:46:00.202534 6123 raft_consensus.cc:348] T be40862863f44123916c80a1ae5b24a9 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:00.202651 6123 raft_consensus.cc:374] T be40862863f44123916c80a1ae5b24a9 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:00.202680 6123 raft_consensus.cc:729] T be40862863f44123916c80a1ae5b24a9 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:46:00.202700 6127 raft_consensus.cc:348] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:00.202816 6127 raft_consensus.cc:374] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:00.202858 6127 raft_consensus.cc:729] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:46:00.202844 6123 consensus_queue.cc:260] T be40862863f44123916c80a1ae5b24a9 P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:00.202989 6127 consensus_queue.cc:260] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:00.203284 6129 tablet_bootstrap.cc:492] T f352efa74d3c4da3907efdc37237211f P 923cfab926e7443f8ffd7b21ad0a9d91: No bootstrap required, opened a new log
I20240502 20:46:00.203362 6129 ts_tablet_manager.cc:1367] T f352efa74d3c4da3907efdc37237211f P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:46:00.203799 6129 raft_consensus.cc:348] T f352efa74d3c4da3907efdc37237211f P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:00.203908 6129 raft_consensus.cc:374] T f352efa74d3c4da3907efdc37237211f P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:00.203948 6129 raft_consensus.cc:729] T f352efa74d3c4da3907efdc37237211f P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: FOLLOWER
I20240502 20:46:00.204048 6129 consensus_queue.cc:260] T f352efa74d3c4da3907efdc37237211f P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:00.204365 6129 ts_tablet_manager.cc:1398] T f352efa74d3c4da3907efdc37237211f P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:00.204524 6129 tablet_bootstrap.cc:492] T be40862863f44123916c80a1ae5b24a9 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:46:00.205458 6129 tablet_bootstrap.cc:654] T be40862863f44123916c80a1ae5b24a9 P 923cfab926e7443f8ffd7b21ad0a9d91: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:00.205502 6127 ts_tablet_manager.cc:1398] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 20:46:00.206722 6129 tablet_bootstrap.cc:492] T be40862863f44123916c80a1ae5b24a9 P 923cfab926e7443f8ffd7b21ad0a9d91: No bootstrap required, opened a new log
I20240502 20:46:00.206804 6129 ts_tablet_manager.cc:1367] T be40862863f44123916c80a1ae5b24a9 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:00.207237 6129 raft_consensus.cc:348] T be40862863f44123916c80a1ae5b24a9 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:00.207374 6129 raft_consensus.cc:374] T be40862863f44123916c80a1ae5b24a9 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:00.207420 6129 raft_consensus.cc:729] T be40862863f44123916c80a1ae5b24a9 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: FOLLOWER
I20240502 20:46:00.207532 6129 consensus_queue.cc:260] T be40862863f44123916c80a1ae5b24a9 P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:00.207800 6129 ts_tablet_manager.cc:1398] T be40862863f44123916c80a1ae5b24a9 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:00.210059 6123 ts_tablet_manager.cc:1398] T be40862863f44123916c80a1ae5b24a9 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240502 20:46:00.210310 6123 tablet_bootstrap.cc:492] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:46:00.212296 6123 tablet_bootstrap.cc:654] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:00.213864 6124 tablet_bootstrap.cc:492] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:46:00.213971 6124 ts_tablet_manager.cc:1367] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240502 20:46:00.214419 6124 raft_consensus.cc:348] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:00.214521 6124 raft_consensus.cc:374] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:00.214548 6124 raft_consensus.cc:729] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:46:00.214656 6124 consensus_queue.cc:260] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:00.214903 6124 ts_tablet_manager.cc:1398] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:00.215049 6124 tablet_bootstrap.cc:492] T f352efa74d3c4da3907efdc37237211f P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:46:00.215958 6124 tablet_bootstrap.cc:654] T f352efa74d3c4da3907efdc37237211f P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:00.217303 6124 tablet_bootstrap.cc:492] T f352efa74d3c4da3907efdc37237211f P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:46:00.217451 6124 ts_tablet_manager.cc:1367] T f352efa74d3c4da3907efdc37237211f P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 20:46:00.217875 6124 raft_consensus.cc:348] T f352efa74d3c4da3907efdc37237211f P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:00.217991 6124 raft_consensus.cc:374] T f352efa74d3c4da3907efdc37237211f P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:00.218021 6124 raft_consensus.cc:729] T f352efa74d3c4da3907efdc37237211f P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:46:00.218178 6124 consensus_queue.cc:260] T f352efa74d3c4da3907efdc37237211f P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:00.218495 6124 ts_tablet_manager.cc:1398] T f352efa74d3c4da3907efdc37237211f P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:00.227885 6123 tablet_bootstrap.cc:492] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:46:00.228020 6123 ts_tablet_manager.cc:1367] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.000s
I20240502 20:46:00.228519 6123 raft_consensus.cc:348] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:00.228647 6123 raft_consensus.cc:374] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:00.228684 6123 raft_consensus.cc:729] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:46:00.228816 6123 consensus_queue.cc:260] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:00.229151 6123 ts_tablet_manager.cc:1398] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240502 20:46:00.229441 6123 tablet_bootstrap.cc:492] T f352efa74d3c4da3907efdc37237211f P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:46:00.229719 5380 raft_consensus.cc:482] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:00.229826 5380 raft_consensus.cc:504] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:00.230311 5380 leader_election.cc:288] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097), c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:46:00.230825 4376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6faa71d4bd2143b0a72cd86a7e1e7e12" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2" is_pre_election: true
I20240502 20:46:00.230983 4376 raft_consensus.cc:2454] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 0.
I20240502 20:46:00.231232 6123 tablet_bootstrap.cc:654] T f352efa74d3c4da3907efdc37237211f P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:00.231387 4706 leader_election.cc:302] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:46:00.232003 4506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6faa71d4bd2143b0a72cd86a7e1e7e12" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" is_pre_election: true
I20240502 20:46:00.232146 4506 raft_consensus.cc:2454] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 0.
I20240502 20:46:00.232479 6123 tablet_bootstrap.cc:492] T f352efa74d3c4da3907efdc37237211f P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:46:00.232558 6123 ts_tablet_manager.cc:1367] T f352efa74d3c4da3907efdc37237211f P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 20:46:00.233048 5380 raft_consensus.cc:2781] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:00.233112 5380 raft_consensus.cc:482] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:00.233067 6123 raft_consensus.cc:348] T f352efa74d3c4da3907efdc37237211f P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:00.233156 5380 raft_consensus.cc:3037] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:00.233165 6123 raft_consensus.cc:374] T f352efa74d3c4da3907efdc37237211f P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:00.233196 6123 raft_consensus.cc:729] T f352efa74d3c4da3907efdc37237211f P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:46:00.233336 6123 consensus_queue.cc:260] T f352efa74d3c4da3907efdc37237211f P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:00.233613 6123 ts_tablet_manager.cc:1398] T f352efa74d3c4da3907efdc37237211f P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:00.234138 5380 raft_consensus.cc:504] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:00.234530 5380 leader_election.cc:288] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 election: Requested vote from peers c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097), c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:46:00.235047 4506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6faa71d4bd2143b0a72cd86a7e1e7e12" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc"
I20240502 20:46:00.235051 4376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6faa71d4bd2143b0a72cd86a7e1e7e12" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2"
I20240502 20:46:00.235172 4506 raft_consensus.cc:3037] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:00.235174 4376 raft_consensus.cc:3037] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:00.236187 4376 raft_consensus.cc:2454] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 1.
I20240502 20:46:00.236186 4506 raft_consensus.cc:2454] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 1.
I20240502 20:46:00.236510 4705 leader_election.cc:302] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3d26eafd51547acb780d4aac5ebe7cc, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:46:00.236727 5380 raft_consensus.cc:2781] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:00.236841 5380 raft_consensus.cc:686] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Becoming Leader. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Running, Role: LEADER
I20240502 20:46:00.237012 5380 consensus_queue.cc:237] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:00.238852 4232 catalog_manager.cc:5484] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P ed47c023452a497192f35d2f617bf9bd reported cstate change: term changed from 0 to 1, leader changed from <none> to ed47c023452a497192f35d2f617bf9bd (127.30.32.68). New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:00.298230 5380 raft_consensus.cc:482] T f352efa74d3c4da3907efdc37237211f P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:00.298364 5380 raft_consensus.cc:504] T f352efa74d3c4da3907efdc37237211f P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:00.298776 5380 leader_election.cc:288] T f352efa74d3c4da3907efdc37237211f P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141), c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097)
I20240502 20:46:00.299034 4638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f352efa74d3c4da3907efdc37237211f" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" is_pre_election: true
I20240502 20:46:00.299163 4638 raft_consensus.cc:2454] T f352efa74d3c4da3907efdc37237211f P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 0.
I20240502 20:46:00.299137 4507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f352efa74d3c4da3907efdc37237211f" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" is_pre_election: true
I20240502 20:46:00.299284 4507 raft_consensus.cc:2454] T f352efa74d3c4da3907efdc37237211f P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 0.
I20240502 20:46:00.299518 4703 leader_election.cc:302] T f352efa74d3c4da3907efdc37237211f P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, c3d26eafd51547acb780d4aac5ebe7cc, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:46:00.299686 5373 raft_consensus.cc:2781] T f352efa74d3c4da3907efdc37237211f P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:00.299764 5373 raft_consensus.cc:482] T f352efa74d3c4da3907efdc37237211f P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:00.299815 5373 raft_consensus.cc:3037] T f352efa74d3c4da3907efdc37237211f P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:00.300673 5373 raft_consensus.cc:504] T f352efa74d3c4da3907efdc37237211f P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:00.301074 5373 leader_election.cc:288] T f352efa74d3c4da3907efdc37237211f P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 election: Requested vote from peers 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141), c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097)
I20240502 20:46:00.301352 4638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f352efa74d3c4da3907efdc37237211f" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91"
I20240502 20:46:00.301445 4507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f352efa74d3c4da3907efdc37237211f" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc"
I20240502 20:46:00.301489 4638 raft_consensus.cc:3037] T f352efa74d3c4da3907efdc37237211f P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:00.301569 4507 raft_consensus.cc:3037] T f352efa74d3c4da3907efdc37237211f P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:00.302289 4638 raft_consensus.cc:2454] T f352efa74d3c4da3907efdc37237211f P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 1.
I20240502 20:46:00.302482 4507 raft_consensus.cc:2454] T f352efa74d3c4da3907efdc37237211f P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 1.
I20240502 20:46:00.302592 4703 leader_election.cc:302] T f352efa74d3c4da3907efdc37237211f P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:46:00.302865 5373 raft_consensus.cc:2781] T f352efa74d3c4da3907efdc37237211f P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:00.302948 5373 raft_consensus.cc:686] T f352efa74d3c4da3907efdc37237211f P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Becoming Leader. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Running, Role: LEADER
I20240502 20:46:00.303079 5373 consensus_queue.cc:237] T f352efa74d3c4da3907efdc37237211f P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:00.305179 4232 catalog_manager.cc:5484] T f352efa74d3c4da3907efdc37237211f P ed47c023452a497192f35d2f617bf9bd reported cstate change: term changed from 0 to 1, leader changed from <none> to ed47c023452a497192f35d2f617bf9bd (127.30.32.68). New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:00.363329 5915 raft_consensus.cc:482] T be40862863f44123916c80a1ae5b24a9 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:00.363490 5915 raft_consensus.cc:504] T be40862863f44123916c80a1ae5b24a9 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:00.363960 5915 leader_election.cc:288] T be40862863f44123916c80a1ae5b24a9 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097), 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:46:00.364286 4506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be40862863f44123916c80a1ae5b24a9" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" is_pre_election: true
I20240502 20:46:00.364377 4638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be40862863f44123916c80a1ae5b24a9" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" is_pre_election: true
I20240502 20:46:00.364427 4506 raft_consensus.cc:2454] T be40862863f44123916c80a1ae5b24a9 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 0.
I20240502 20:46:00.364508 4638 raft_consensus.cc:2454] T be40862863f44123916c80a1ae5b24a9 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 0.
I20240502 20:46:00.364789 4312 leader_election.cc:302] T be40862863f44123916c80a1ae5b24a9 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3d26eafd51547acb780d4aac5ebe7cc, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:46:00.365005 5915 raft_consensus.cc:2781] T be40862863f44123916c80a1ae5b24a9 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:00.365067 5915 raft_consensus.cc:482] T be40862863f44123916c80a1ae5b24a9 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:00.365113 5915 raft_consensus.cc:3037] T be40862863f44123916c80a1ae5b24a9 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:00.365985 5915 raft_consensus.cc:504] T be40862863f44123916c80a1ae5b24a9 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:00.366312 5915 leader_election.cc:288] T be40862863f44123916c80a1ae5b24a9 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 election: Requested vote from peers c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097), 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:46:00.366667 4638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be40862863f44123916c80a1ae5b24a9" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91"
I20240502 20:46:00.366681 4506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be40862863f44123916c80a1ae5b24a9" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc"
I20240502 20:46:00.366820 4506 raft_consensus.cc:3037] T be40862863f44123916c80a1ae5b24a9 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:00.366837 4638 raft_consensus.cc:3037] T be40862863f44123916c80a1ae5b24a9 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:00.367790 4506 raft_consensus.cc:2454] T be40862863f44123916c80a1ae5b24a9 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 1.
I20240502 20:46:00.367798 4638 raft_consensus.cc:2454] T be40862863f44123916c80a1ae5b24a9 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 1.
I20240502 20:46:00.368052 4312 leader_election.cc:302] T be40862863f44123916c80a1ae5b24a9 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3d26eafd51547acb780d4aac5ebe7cc, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:46:00.368237 5915 raft_consensus.cc:2781] T be40862863f44123916c80a1ae5b24a9 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:00.368439 5915 raft_consensus.cc:686] T be40862863f44123916c80a1ae5b24a9 P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEADER]: Becoming Leader. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Running, Role: LEADER
I20240502 20:46:00.368597 5915 consensus_queue.cc:237] T be40862863f44123916c80a1ae5b24a9 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:00.370492 4232 catalog_manager.cc:5484] T be40862863f44123916c80a1ae5b24a9 P c92e6e46782849479c8a0c6d82276fb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65). New cstate: current_term: 1 leader_uuid: "c92e6e46782849479c8a0c6d82276fb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: HEALTHY } } }
W20240502 20:46:00.510120 6140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:00.510377 6140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:00.513447 6140 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_12'
I20240502 20:46:00.620189 5373 consensus_queue.cc:1035] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:00.627292 4638 raft_consensus.cc:1261] T be40862863f44123916c80a1ae5b24a9 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:00.627328 4506 raft_consensus.cc:1261] T f352efa74d3c4da3907efdc37237211f P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:00.627385 4507 raft_consensus.cc:1261] T be40862863f44123916c80a1ae5b24a9 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:00.627746 5915 consensus_queue.cc:1035] T be40862863f44123916c80a1ae5b24a9 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:00.627842 5373 consensus_queue.cc:1035] T f352efa74d3c4da3907efdc37237211f P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:00.628031 5915 consensus_queue.cc:1035] T be40862863f44123916c80a1ae5b24a9 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:00.631000 4376 raft_consensus.cc:1261] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:00.633173 5373 consensus_queue.cc:1035] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:00.636265 4639 raft_consensus.cc:1261] T f352efa74d3c4da3907efdc37237211f P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:00.636731 6177 consensus_queue.cc:1035] T f352efa74d3c4da3907efdc37237211f P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 145.359 ms
time per row: 0.0726797 ms
I20240502 20:46:00.699611 4229 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41812:
name: "rebalance_test_table_13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:46:00.705116 4355 tablet_service.cc:1467] Processing CreateTablet for tablet c9d25657e9a14b35b2f6e8d82f4533c7 (DEFAULT_TABLE table=rebalance_test_table_13 [id=c22b8adbea7b4dc49a4794a5d8f81f9b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:00.705148 4618 tablet_service.cc:1467] Processing CreateTablet for tablet c9d25657e9a14b35b2f6e8d82f4533c7 (DEFAULT_TABLE table=rebalance_test_table_13 [id=c22b8adbea7b4dc49a4794a5d8f81f9b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:00.705433 4618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9d25657e9a14b35b2f6e8d82f4533c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:00.705452 4355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9d25657e9a14b35b2f6e8d82f4533c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:00.707545 6129 tablet_bootstrap.cc:492] T c9d25657e9a14b35b2f6e8d82f4533c7 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:46:00.707717 6127 tablet_bootstrap.cc:492] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:46:00.708024 4750 tablet_service.cc:1467] Processing CreateTablet for tablet c9d25657e9a14b35b2f6e8d82f4533c7 (DEFAULT_TABLE table=rebalance_test_table_13 [id=c22b8adbea7b4dc49a4794a5d8f81f9b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:00.708040 4749 tablet_service.cc:1467] Processing CreateTablet for tablet 7abfd61f55554042978b50761cbd6ab5 (DEFAULT_TABLE table=rebalance_test_table_13 [id=c22b8adbea7b4dc49a4794a5d8f81f9b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:00.708320 4750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9d25657e9a14b35b2f6e8d82f4533c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:00.708460 4749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7abfd61f55554042978b50761cbd6ab5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:00.708467 6129 tablet_bootstrap.cc:654] T c9d25657e9a14b35b2f6e8d82f4533c7 P 923cfab926e7443f8ffd7b21ad0a9d91: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:00.708532 6127 tablet_bootstrap.cc:654] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:00.709419 4357 tablet_service.cc:1467] Processing CreateTablet for tablet 41f84d5db2034af591b56e109be67934 (DEFAULT_TABLE table=rebalance_test_table_13 [id=c22b8adbea7b4dc49a4794a5d8f81f9b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:00.709601 6127 tablet_bootstrap.cc:492] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:46:00.709648 4357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41f84d5db2034af591b56e109be67934. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:00.709682 6127 ts_tablet_manager.cc:1367] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 20:46:00.710117 6127 raft_consensus.cc:348] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:00.710214 6127 raft_consensus.cc:374] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:00.710254 6127 raft_consensus.cc:729] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:46:00.710377 6127 consensus_queue.cc:260] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:00.710686 6127 ts_tablet_manager.cc:1398] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:00.710749 6124 tablet_bootstrap.cc:492] T 7abfd61f55554042978b50761cbd6ab5 P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:46:00.711508 6124 tablet_bootstrap.cc:654] T 7abfd61f55554042978b50761cbd6ab5 P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:00.711762 4487 tablet_service.cc:1467] Processing CreateTablet for tablet 41f84d5db2034af591b56e109be67934 (DEFAULT_TABLE table=rebalance_test_table_13 [id=c22b8adbea7b4dc49a4794a5d8f81f9b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:00.712036 4487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41f84d5db2034af591b56e109be67934. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:00.712165 4617 tablet_service.cc:1467] Processing CreateTablet for tablet 41f84d5db2034af591b56e109be67934 (DEFAULT_TABLE table=rebalance_test_table_13 [id=c22b8adbea7b4dc49a4794a5d8f81f9b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:00.712395 4617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41f84d5db2034af591b56e109be67934. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:00.713284 4618 tablet_service.cc:1467] Processing CreateTablet for tablet 7abfd61f55554042978b50761cbd6ab5 (DEFAULT_TABLE table=rebalance_test_table_13 [id=c22b8adbea7b4dc49a4794a5d8f81f9b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:00.713541 4618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7abfd61f55554042978b50761cbd6ab5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:00.714032 6123 tablet_bootstrap.cc:492] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:46:00.714744 6127 tablet_bootstrap.cc:492] T 41f84d5db2034af591b56e109be67934 P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:46:00.714789 6123 tablet_bootstrap.cc:654] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:00.715755 6127 tablet_bootstrap.cc:654] T 41f84d5db2034af591b56e109be67934 P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:00.716143 6123 tablet_bootstrap.cc:492] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:46:00.716228 6123 ts_tablet_manager.cc:1367] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:00.716321 4355 tablet_service.cc:1467] Processing CreateTablet for tablet 7abfd61f55554042978b50761cbd6ab5 (DEFAULT_TABLE table=rebalance_test_table_13 [id=c22b8adbea7b4dc49a4794a5d8f81f9b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:00.716553 4355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7abfd61f55554042978b50761cbd6ab5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:00.716678 6123 raft_consensus.cc:348] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:00.716779 6123 raft_consensus.cc:374] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:00.716820 6123 raft_consensus.cc:729] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:46:00.716953 6123 consensus_queue.cc:260] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:00.717244 6123 ts_tablet_manager.cc:1398] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:00.720125 6124 tablet_bootstrap.cc:492] T 7abfd61f55554042978b50761cbd6ab5 P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:46:00.720227 6124 ts_tablet_manager.cc:1367] T 7abfd61f55554042978b50761cbd6ab5 P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20240502 20:46:00.720700 6124 raft_consensus.cc:348] T 7abfd61f55554042978b50761cbd6ab5 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:00.720819 6124 raft_consensus.cc:374] T 7abfd61f55554042978b50761cbd6ab5 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:00.720852 6124 raft_consensus.cc:729] T 7abfd61f55554042978b50761cbd6ab5 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:46:00.720971 6124 consensus_queue.cc:260] T 7abfd61f55554042978b50761cbd6ab5 P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:00.721068 6127 tablet_bootstrap.cc:492] T 41f84d5db2034af591b56e109be67934 P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:46:00.721161 6127 ts_tablet_manager.cc:1367] T 41f84d5db2034af591b56e109be67934 P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240502 20:46:00.721302 6124 ts_tablet_manager.cc:1398] T 7abfd61f55554042978b50761cbd6ab5 P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:00.721441 6124 tablet_bootstrap.cc:492] T c9d25657e9a14b35b2f6e8d82f4533c7 P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:46:00.721593 6127 raft_consensus.cc:348] T 41f84d5db2034af591b56e109be67934 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:00.721689 6127 raft_consensus.cc:374] T 41f84d5db2034af591b56e109be67934 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:00.721721 6127 raft_consensus.cc:729] T 41f84d5db2034af591b56e109be67934 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:46:00.721838 6127 consensus_queue.cc:260] T 41f84d5db2034af591b56e109be67934 P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:00.722147 6127 ts_tablet_manager.cc:1398] T 41f84d5db2034af591b56e109be67934 P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:00.722235 6124 tablet_bootstrap.cc:654] T c9d25657e9a14b35b2f6e8d82f4533c7 P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:00.722402 6127 tablet_bootstrap.cc:492] T 7abfd61f55554042978b50761cbd6ab5 P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:46:00.723246 6124 tablet_bootstrap.cc:492] T c9d25657e9a14b35b2f6e8d82f4533c7 P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:46:00.723315 6127 tablet_bootstrap.cc:654] T 7abfd61f55554042978b50761cbd6ab5 P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:00.723333 6124 ts_tablet_manager.cc:1367] T c9d25657e9a14b35b2f6e8d82f4533c7 P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 20:46:00.723740 6124 raft_consensus.cc:348] T c9d25657e9a14b35b2f6e8d82f4533c7 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:00.723930 6124 raft_consensus.cc:374] T c9d25657e9a14b35b2f6e8d82f4533c7 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:00.723973 6124 raft_consensus.cc:729] T c9d25657e9a14b35b2f6e8d82f4533c7 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:46:00.724103 6124 consensus_queue.cc:260] T c9d25657e9a14b35b2f6e8d82f4533c7 P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:00.724479 6124 ts_tablet_manager.cc:1398] T c9d25657e9a14b35b2f6e8d82f4533c7 P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240502 20:46:00.724609 6127 tablet_bootstrap.cc:492] T 7abfd61f55554042978b50761cbd6ab5 P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:46:00.724673 6127 ts_tablet_manager.cc:1367] T 7abfd61f55554042978b50761cbd6ab5 P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 20:46:00.725500 6127 raft_consensus.cc:348] T 7abfd61f55554042978b50761cbd6ab5 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:00.725576 6129 tablet_bootstrap.cc:492] T c9d25657e9a14b35b2f6e8d82f4533c7 P 923cfab926e7443f8ffd7b21ad0a9d91: No bootstrap required, opened a new log
I20240502 20:46:00.725625 6127 raft_consensus.cc:374] T 7abfd61f55554042978b50761cbd6ab5 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:00.725672 6127 raft_consensus.cc:729] T 7abfd61f55554042978b50761cbd6ab5 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:46:00.725677 6129 ts_tablet_manager.cc:1367] T c9d25657e9a14b35b2f6e8d82f4533c7 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240502 20:46:00.725803 6127 consensus_queue.cc:260] T 7abfd61f55554042978b50761cbd6ab5 P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:00.726050 6127 ts_tablet_manager.cc:1398] T 7abfd61f55554042978b50761cbd6ab5 P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:00.726153 6129 raft_consensus.cc:348] T c9d25657e9a14b35b2f6e8d82f4533c7 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:00.726301 6129 raft_consensus.cc:374] T c9d25657e9a14b35b2f6e8d82f4533c7 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:00.726353 6129 raft_consensus.cc:729] T c9d25657e9a14b35b2f6e8d82f4533c7 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: FOLLOWER
I20240502 20:46:00.726480 6129 consensus_queue.cc:260] T c9d25657e9a14b35b2f6e8d82f4533c7 P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:00.726785 6129 ts_tablet_manager.cc:1398] T c9d25657e9a14b35b2f6e8d82f4533c7 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:00.726972 6129 tablet_bootstrap.cc:492] T 41f84d5db2034af591b56e109be67934 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:46:00.727823 6129 tablet_bootstrap.cc:654] T 41f84d5db2034af591b56e109be67934 P 923cfab926e7443f8ffd7b21ad0a9d91: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:00.729398 6129 tablet_bootstrap.cc:492] T 41f84d5db2034af591b56e109be67934 P 923cfab926e7443f8ffd7b21ad0a9d91: No bootstrap required, opened a new log
I20240502 20:46:00.729497 6129 ts_tablet_manager.cc:1367] T 41f84d5db2034af591b56e109be67934 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 20:46:00.729938 6129 raft_consensus.cc:348] T 41f84d5db2034af591b56e109be67934 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:00.730118 6129 raft_consensus.cc:374] T 41f84d5db2034af591b56e109be67934 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:00.730155 6129 raft_consensus.cc:729] T 41f84d5db2034af591b56e109be67934 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: FOLLOWER
I20240502 20:46:00.730280 6129 consensus_queue.cc:260] T 41f84d5db2034af591b56e109be67934 P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:00.730594 6129 ts_tablet_manager.cc:1398] T 41f84d5db2034af591b56e109be67934 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:00.730765 6129 tablet_bootstrap.cc:492] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:46:00.732962 6129 tablet_bootstrap.cc:654] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:00.734017 6129 tablet_bootstrap.cc:492] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91: No bootstrap required, opened a new log
I20240502 20:46:00.734088 6129 ts_tablet_manager.cc:1367] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 20:46:00.734396 6129 raft_consensus.cc:348] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:00.734477 6129 raft_consensus.cc:374] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:00.734512 6129 raft_consensus.cc:729] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: FOLLOWER
I20240502 20:46:00.734611 6129 consensus_queue.cc:260] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:00.734871 6129 ts_tablet_manager.cc:1398] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:00.752190 6036 raft_consensus.cc:482] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:00.752342 6036 raft_consensus.cc:504] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:00.752781 6036 leader_election.cc:288] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827), 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:46:00.753125 4638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41f84d5db2034af591b56e109be67934" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" is_pre_election: true
I20240502 20:46:00.753151 4376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41f84d5db2034af591b56e109be67934" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2" is_pre_election: true
I20240502 20:46:00.753242 4638 raft_consensus.cc:2454] T 41f84d5db2034af591b56e109be67934 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 0.
I20240502 20:46:00.753295 4376 raft_consensus.cc:2454] T 41f84d5db2034af591b56e109be67934 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 0.
I20240502 20:46:00.753556 4441 leader_election.cc:302] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, c3d26eafd51547acb780d4aac5ebe7cc; no voters:
I20240502 20:46:00.753813 6036 raft_consensus.cc:2781] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:00.753875 6036 raft_consensus.cc:482] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:00.753916 6036 raft_consensus.cc:3037] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:00.755039 6036 raft_consensus.cc:504] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:00.755405 6036 leader_election.cc:288] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 election: Requested vote from peers c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827), 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:46:00.755697 4376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41f84d5db2034af591b56e109be67934" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2"
I20240502 20:46:00.755760 4638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41f84d5db2034af591b56e109be67934" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91"
I20240502 20:46:00.755825 4376 raft_consensus.cc:3037] T 41f84d5db2034af591b56e109be67934 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:00.755874 4638 raft_consensus.cc:3037] T 41f84d5db2034af591b56e109be67934 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:00.756793 4638 raft_consensus.cc:2454] T 41f84d5db2034af591b56e109be67934 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 1.
I20240502 20:46:00.756847 4376 raft_consensus.cc:2454] T 41f84d5db2034af591b56e109be67934 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 1.
I20240502 20:46:00.757107 4441 leader_election.cc:302] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, c3d26eafd51547acb780d4aac5ebe7cc; no voters:
I20240502 20:46:00.757303 6036 raft_consensus.cc:2781] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:00.757395 6036 raft_consensus.cc:686] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Becoming Leader. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Running, Role: LEADER
I20240502 20:46:00.757535 6036 consensus_queue.cc:237] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:00.759434 4229 catalog_manager.cc:5484] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: term changed from 0 to 1, leader changed from <none> to c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66). New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:00.824813 6098 raft_consensus.cc:482] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:00.824954 6098 raft_consensus.cc:504] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:00.825377 6098 leader_election.cc:288] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827), ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:46:00.825873 4376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7abfd61f55554042978b50761cbd6ab5" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2" is_pre_election: true
I20240502 20:46:00.825937 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7abfd61f55554042978b50761cbd6ab5" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd" is_pre_election: true
I20240502 20:46:00.826043 4376 raft_consensus.cc:2454] T 7abfd61f55554042978b50761cbd6ab5 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 0.
I20240502 20:46:00.826100 4768 raft_consensus.cc:2454] T 7abfd61f55554042978b50761cbd6ab5 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 0.
I20240502 20:46:00.826445 4575 leader_election.cc:302] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:46:00.826596 6098 raft_consensus.cc:2781] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:00.826673 6098 raft_consensus.cc:482] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:00.826740 6098 raft_consensus.cc:3037] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:00.827606 6098 raft_consensus.cc:504] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:00.827948 6098 leader_election.cc:288] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 election: Requested vote from peers c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827), ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:46:00.828341 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7abfd61f55554042978b50761cbd6ab5" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd"
I20240502 20:46:00.828351 4376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7abfd61f55554042978b50761cbd6ab5" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2"
I20240502 20:46:00.828449 4768 raft_consensus.cc:3037] T 7abfd61f55554042978b50761cbd6ab5 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:00.828464 4376 raft_consensus.cc:3037] T 7abfd61f55554042978b50761cbd6ab5 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:00.829231 4376 raft_consensus.cc:2454] T 7abfd61f55554042978b50761cbd6ab5 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 1.
I20240502 20:46:00.829274 4768 raft_consensus.cc:2454] T 7abfd61f55554042978b50761cbd6ab5 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 1.
I20240502 20:46:00.829519 4575 leader_election.cc:302] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:46:00.829699 6098 raft_consensus.cc:2781] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:00.829793 6098 raft_consensus.cc:686] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Becoming Leader. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Running, Role: LEADER
I20240502 20:46:00.829942 6098 consensus_queue.cc:237] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:00.831840 4229 catalog_manager.cc:5484] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67). New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:00.959925 5915 raft_consensus.cc:482] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:00.960083 5915 raft_consensus.cc:504] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:00.960566 5915 leader_election.cc:288] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193), 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:46:00.961071 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9d25657e9a14b35b2f6e8d82f4533c7" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd" is_pre_election: true
I20240502 20:46:00.961074 4638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9d25657e9a14b35b2f6e8d82f4533c7" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" is_pre_election: true
I20240502 20:46:00.961215 4768 raft_consensus.cc:2454] T c9d25657e9a14b35b2f6e8d82f4533c7 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 0.
I20240502 20:46:00.961230 4638 raft_consensus.cc:2454] T c9d25657e9a14b35b2f6e8d82f4533c7 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 0.
I20240502 20:46:00.961548 4313 leader_election.cc:302] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:46:00.961721 5915 raft_consensus.cc:2781] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:00.961782 5915 raft_consensus.cc:482] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:00.961831 5915 raft_consensus.cc:3037] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:00.962657 5915 raft_consensus.cc:504] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:00.962982 5915 leader_election.cc:288] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 election: Requested vote from peers ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193), 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:46:00.963464 4638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9d25657e9a14b35b2f6e8d82f4533c7" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91"
I20240502 20:46:00.963637 4638 raft_consensus.cc:3037] T c9d25657e9a14b35b2f6e8d82f4533c7 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:00.964252 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9d25657e9a14b35b2f6e8d82f4533c7" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd"
I20240502 20:46:00.964357 4768 raft_consensus.cc:3037] T c9d25657e9a14b35b2f6e8d82f4533c7 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:00.964613 4638 raft_consensus.cc:2454] T c9d25657e9a14b35b2f6e8d82f4533c7 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 1.
I20240502 20:46:00.964915 4310 leader_election.cc:302] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:46:00.965126 5915 raft_consensus.cc:2781] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:00.965220 5915 raft_consensus.cc:686] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEADER]: Becoming Leader. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Running, Role: LEADER
I20240502 20:46:00.965363 4768 raft_consensus.cc:2454] T c9d25657e9a14b35b2f6e8d82f4533c7 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 1.
I20240502 20:46:00.965392 5915 consensus_queue.cc:237] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:00.967710 4229 catalog_manager.cc:5484] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65). New cstate: current_term: 1 leader_uuid: "c92e6e46782849479c8a0c6d82276fb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: UNKNOWN } } }
W20240502 20:46:01.123522 6198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:01.123785 6198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:01.127054 6198 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_13'
I20240502 20:46:01.176517 6036 consensus_queue.cc:1035] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:01.181954 6037 consensus_queue.cc:1035] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:01.208814 4638 raft_consensus.cc:1261] T c9d25657e9a14b35b2f6e8d82f4533c7 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:01.208814 4770 raft_consensus.cc:1261] T c9d25657e9a14b35b2f6e8d82f4533c7 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:01.209283 6138 consensus_queue.cc:1035] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:01.209439 5915 consensus_queue.cc:1035] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:01.225843 4376 raft_consensus.cc:1261] T 7abfd61f55554042978b50761cbd6ab5 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:01.225869 4770 raft_consensus.cc:1261] T 7abfd61f55554042978b50761cbd6ab5 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:01.226325 6195 consensus_queue.cc:1035] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:01.226617 6195 consensus_queue.cc:1035] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 127.678 ms
time per row: 0.0638388 ms
I20240502 20:46:01.297945 4229 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41812:
name: "rebalance_test_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:46:01.303834 4750 tablet_service.cc:1467] Processing CreateTablet for tablet e5a4dc24914c4f37a14481b77c4fcdaa (DEFAULT_TABLE table=rebalance_test_table_14 [id=a699ad9359e6422b8223c2844e565c67]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:01.303926 4487 tablet_service.cc:1467] Processing CreateTablet for tablet e5a4dc24914c4f37a14481b77c4fcdaa (DEFAULT_TABLE table=rebalance_test_table_14 [id=a699ad9359e6422b8223c2844e565c67]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:01.304067 4355 tablet_service.cc:1467] Processing CreateTablet for tablet e5a4dc24914c4f37a14481b77c4fcdaa (DEFAULT_TABLE table=rebalance_test_table_14 [id=a699ad9359e6422b8223c2844e565c67]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:01.304183 4750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5a4dc24914c4f37a14481b77c4fcdaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:01.304246 4487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5a4dc24914c4f37a14481b77c4fcdaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:01.304348 4355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5a4dc24914c4f37a14481b77c4fcdaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:01.304941 4488 tablet_service.cc:1467] Processing CreateTablet for tablet 04c72d7d63574dea99cb2f7c14199a73 (DEFAULT_TABLE table=rebalance_test_table_14 [id=a699ad9359e6422b8223c2844e565c67]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:01.305191 4488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04c72d7d63574dea99cb2f7c14199a73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:01.306906 4486 tablet_service.cc:1467] Processing CreateTablet for tablet 6c9adf5004a649979b9abb97df5c0c28 (DEFAULT_TABLE table=rebalance_test_table_14 [id=a699ad9359e6422b8223c2844e565c67]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:01.307126 6258 tablet_bootstrap.cc:492] T e5a4dc24914c4f37a14481b77c4fcdaa P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:46:01.307147 4486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c9adf5004a649979b9abb97df5c0c28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:01.307864 4618 tablet_service.cc:1467] Processing CreateTablet for tablet 6c9adf5004a649979b9abb97df5c0c28 (DEFAULT_TABLE table=rebalance_test_table_14 [id=a699ad9359e6422b8223c2844e565c67]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:01.308166 4618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c9adf5004a649979b9abb97df5c0c28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:01.308244 4750 tablet_service.cc:1467] Processing CreateTablet for tablet 6c9adf5004a649979b9abb97df5c0c28 (DEFAULT_TABLE table=rebalance_test_table_14 [id=a699ad9359e6422b8223c2844e565c67]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:01.308458 4750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c9adf5004a649979b9abb97df5c0c28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:01.308583 6259 tablet_bootstrap.cc:492] T e5a4dc24914c4f37a14481b77c4fcdaa P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:46:01.309218 4749 tablet_service.cc:1467] Processing CreateTablet for tablet 04c72d7d63574dea99cb2f7c14199a73 (DEFAULT_TABLE table=rebalance_test_table_14 [id=a699ad9359e6422b8223c2844e565c67]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:01.309453 4749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04c72d7d63574dea99cb2f7c14199a73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:01.309523 6259 tablet_bootstrap.cc:654] T e5a4dc24914c4f37a14481b77c4fcdaa P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:01.310451 6257 tablet_bootstrap.cc:492] T e5a4dc24914c4f37a14481b77c4fcdaa P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:46:01.310698 6258 tablet_bootstrap.cc:654] T e5a4dc24914c4f37a14481b77c4fcdaa P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:01.311215 6257 tablet_bootstrap.cc:654] T e5a4dc24914c4f37a14481b77c4fcdaa P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:01.311775 6259 tablet_bootstrap.cc:492] T e5a4dc24914c4f37a14481b77c4fcdaa P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:46:01.311867 6259 ts_tablet_manager.cc:1367] T e5a4dc24914c4f37a14481b77c4fcdaa P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:46:01.311970 4355 tablet_service.cc:1467] Processing CreateTablet for tablet 04c72d7d63574dea99cb2f7c14199a73 (DEFAULT_TABLE table=rebalance_test_table_14 [id=a699ad9359e6422b8223c2844e565c67]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:01.312175 6258 tablet_bootstrap.cc:492] T e5a4dc24914c4f37a14481b77c4fcdaa P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:46:01.312206 4355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04c72d7d63574dea99cb2f7c14199a73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:01.312280 6258 ts_tablet_manager.cc:1367] T e5a4dc24914c4f37a14481b77c4fcdaa P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240502 20:46:01.312330 6259 raft_consensus.cc:348] T e5a4dc24914c4f37a14481b77c4fcdaa P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:01.312444 6259 raft_consensus.cc:374] T e5a4dc24914c4f37a14481b77c4fcdaa P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:01.312491 6259 raft_consensus.cc:729] T e5a4dc24914c4f37a14481b77c4fcdaa P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:46:01.312669 6259 consensus_queue.cc:260] T e5a4dc24914c4f37a14481b77c4fcdaa P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:01.312717 6258 raft_consensus.cc:348] T e5a4dc24914c4f37a14481b77c4fcdaa P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:01.312820 6258 raft_consensus.cc:374] T e5a4dc24914c4f37a14481b77c4fcdaa P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:01.312860 6258 raft_consensus.cc:729] T e5a4dc24914c4f37a14481b77c4fcdaa P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:46:01.312894 6257 tablet_bootstrap.cc:492] T e5a4dc24914c4f37a14481b77c4fcdaa P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:46:01.313055 6258 consensus_queue.cc:260] T e5a4dc24914c4f37a14481b77c4fcdaa P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:01.313133 6259 ts_tablet_manager.cc:1398] T e5a4dc24914c4f37a14481b77c4fcdaa P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:01.313187 6257 ts_tablet_manager.cc:1367] T e5a4dc24914c4f37a14481b77c4fcdaa P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:46:01.313422 6258 ts_tablet_manager.cc:1398] T e5a4dc24914c4f37a14481b77c4fcdaa P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:01.314288 6257 raft_consensus.cc:348] T e5a4dc24914c4f37a14481b77c4fcdaa P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:01.314399 6257 raft_consensus.cc:374] T e5a4dc24914c4f37a14481b77c4fcdaa P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:01.314432 6257 raft_consensus.cc:729] T e5a4dc24914c4f37a14481b77c4fcdaa P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:46:01.314568 6257 consensus_queue.cc:260] T e5a4dc24914c4f37a14481b77c4fcdaa P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:01.314929 6257 ts_tablet_manager.cc:1398] T e5a4dc24914c4f37a14481b77c4fcdaa P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:01.314968 6259 tablet_bootstrap.cc:492] T 04c72d7d63574dea99cb2f7c14199a73 P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:46:01.315172 6257 tablet_bootstrap.cc:492] T 04c72d7d63574dea99cb2f7c14199a73 P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:46:01.316154 6258 tablet_bootstrap.cc:492] T 6c9adf5004a649979b9abb97df5c0c28 P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:46:01.316684 6257 tablet_bootstrap.cc:654] T 04c72d7d63574dea99cb2f7c14199a73 P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:01.317099 6258 tablet_bootstrap.cc:654] T 6c9adf5004a649979b9abb97df5c0c28 P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:01.317260 6259 tablet_bootstrap.cc:654] T 04c72d7d63574dea99cb2f7c14199a73 P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:01.318421 6261 tablet_bootstrap.cc:492] T 6c9adf5004a649979b9abb97df5c0c28 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:46:01.318491 6259 tablet_bootstrap.cc:492] T 04c72d7d63574dea99cb2f7c14199a73 P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:46:01.318583 6259 ts_tablet_manager.cc:1367] T 04c72d7d63574dea99cb2f7c14199a73 P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 20:46:01.318964 6257 tablet_bootstrap.cc:492] T 04c72d7d63574dea99cb2f7c14199a73 P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:46:01.319046 6257 ts_tablet_manager.cc:1367] T 04c72d7d63574dea99cb2f7c14199a73 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 20:46:01.319047 6259 raft_consensus.cc:348] T 04c72d7d63574dea99cb2f7c14199a73 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:01.319133 6259 raft_consensus.cc:374] T 04c72d7d63574dea99cb2f7c14199a73 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:01.319170 6259 raft_consensus.cc:729] T 04c72d7d63574dea99cb2f7c14199a73 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:46:01.319268 6259 consensus_queue.cc:260] T 04c72d7d63574dea99cb2f7c14199a73 P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:01.319326 6261 tablet_bootstrap.cc:654] T 6c9adf5004a649979b9abb97df5c0c28 P 923cfab926e7443f8ffd7b21ad0a9d91: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:01.319530 6259 ts_tablet_manager.cc:1398] T 04c72d7d63574dea99cb2f7c14199a73 P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:01.319517 6257 raft_consensus.cc:348] T 04c72d7d63574dea99cb2f7c14199a73 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:01.320086 6257 raft_consensus.cc:374] T 04c72d7d63574dea99cb2f7c14199a73 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:01.320147 6257 raft_consensus.cc:729] T 04c72d7d63574dea99cb2f7c14199a73 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:46:01.320387 6257 consensus_queue.cc:260] T 04c72d7d63574dea99cb2f7c14199a73 P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:01.320508 6258 tablet_bootstrap.cc:492] T 6c9adf5004a649979b9abb97df5c0c28 P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:46:01.320583 6258 ts_tablet_manager.cc:1367] T 6c9adf5004a649979b9abb97df5c0c28 P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 20:46:01.320705 6257 ts_tablet_manager.cc:1398] T 04c72d7d63574dea99cb2f7c14199a73 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:01.320883 6257 tablet_bootstrap.cc:492] T 6c9adf5004a649979b9abb97df5c0c28 P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:46:01.321027 6258 raft_consensus.cc:348] T 6c9adf5004a649979b9abb97df5c0c28 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:01.321136 6258 raft_consensus.cc:374] T 6c9adf5004a649979b9abb97df5c0c28 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:01.321169 6258 raft_consensus.cc:729] T 6c9adf5004a649979b9abb97df5c0c28 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:46:01.321296 6258 consensus_queue.cc:260] T 6c9adf5004a649979b9abb97df5c0c28 P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:01.321560 6258 ts_tablet_manager.cc:1398] T 6c9adf5004a649979b9abb97df5c0c28 P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:01.321708 6258 tablet_bootstrap.cc:492] T 04c72d7d63574dea99cb2f7c14199a73 P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:46:01.321827 6257 tablet_bootstrap.cc:654] T 6c9adf5004a649979b9abb97df5c0c28 P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:01.322499 6258 tablet_bootstrap.cc:654] T 04c72d7d63574dea99cb2f7c14199a73 P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:01.323920 6258 tablet_bootstrap.cc:492] T 04c72d7d63574dea99cb2f7c14199a73 P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:46:01.324004 6258 ts_tablet_manager.cc:1367] T 04c72d7d63574dea99cb2f7c14199a73 P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:01.323947 6261 tablet_bootstrap.cc:492] T 6c9adf5004a649979b9abb97df5c0c28 P 923cfab926e7443f8ffd7b21ad0a9d91: No bootstrap required, opened a new log
I20240502 20:46:01.324208 6261 ts_tablet_manager.cc:1367] T 6c9adf5004a649979b9abb97df5c0c28 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 20:46:01.324669 6257 tablet_bootstrap.cc:492] T 6c9adf5004a649979b9abb97df5c0c28 P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:46:01.324661 6258 raft_consensus.cc:348] T 04c72d7d63574dea99cb2f7c14199a73 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:01.324765 6257 ts_tablet_manager.cc:1367] T 6c9adf5004a649979b9abb97df5c0c28 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 20:46:01.324764 6258 raft_consensus.cc:374] T 04c72d7d63574dea99cb2f7c14199a73 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:01.324805 6258 raft_consensus.cc:729] T 04c72d7d63574dea99cb2f7c14199a73 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:46:01.324699 6261 raft_consensus.cc:348] T 6c9adf5004a649979b9abb97df5c0c28 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:01.324927 6258 consensus_queue.cc:260] T 04c72d7d63574dea99cb2f7c14199a73 P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:01.325080 6261 raft_consensus.cc:374] T 6c9adf5004a649979b9abb97df5c0c28 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:01.325186 6261 raft_consensus.cc:729] T 6c9adf5004a649979b9abb97df5c0c28 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: FOLLOWER
I20240502 20:46:01.325263 6258 ts_tablet_manager.cc:1398] T 04c72d7d63574dea99cb2f7c14199a73 P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:01.325250 6257 raft_consensus.cc:348] T 6c9adf5004a649979b9abb97df5c0c28 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:01.325347 6257 raft_consensus.cc:374] T 6c9adf5004a649979b9abb97df5c0c28 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:01.325500 6261 consensus_queue.cc:260] T 6c9adf5004a649979b9abb97df5c0c28 P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:01.325870 6261 ts_tablet_manager.cc:1398] T 6c9adf5004a649979b9abb97df5c0c28 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:01.325392 6257 raft_consensus.cc:729] T 6c9adf5004a649979b9abb97df5c0c28 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:46:01.326027 6257 consensus_queue.cc:260] T 6c9adf5004a649979b9abb97df5c0c28 P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:01.326448 6257 ts_tablet_manager.cc:1398] T 6c9adf5004a649979b9abb97df5c0c28 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:01.354154 6244 raft_consensus.cc:482] T 04c72d7d63574dea99cb2f7c14199a73 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:01.354282 6244 raft_consensus.cc:504] T 04c72d7d63574dea99cb2f7c14199a73 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:01.354624 6244 leader_election.cc:288] T 04c72d7d63574dea99cb2f7c14199a73 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097), ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:46:01.355049 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04c72d7d63574dea99cb2f7c14199a73" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd" is_pre_election: true
I20240502 20:46:01.355048 6238 raft_consensus.cc:482] T 6c9adf5004a649979b9abb97df5c0c28 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:01.355027 4508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04c72d7d63574dea99cb2f7c14199a73" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" is_pre_election: true
I20240502 20:46:01.355156 6238 raft_consensus.cc:504] T 6c9adf5004a649979b9abb97df5c0c28 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:01.355188 4770 raft_consensus.cc:2454] T 04c72d7d63574dea99cb2f7c14199a73 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 0.
I20240502 20:46:01.355185 4508 raft_consensus.cc:2454] T 04c72d7d63574dea99cb2f7c14199a73 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 0.
I20240502 20:46:01.355479 4313 leader_election.cc:302] T 04c72d7d63574dea99cb2f7c14199a73 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:46:01.355787 6244 raft_consensus.cc:2781] T 04c72d7d63574dea99cb2f7c14199a73 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:01.355845 6244 raft_consensus.cc:482] T 04c72d7d63574dea99cb2f7c14199a73 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:01.355890 6244 raft_consensus.cc:3037] T 04c72d7d63574dea99cb2f7c14199a73 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:01.355842 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c9adf5004a649979b9abb97df5c0c28" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd" is_pre_election: true
I20240502 20:46:01.355935 6238 leader_election.cc:288] T 6c9adf5004a649979b9abb97df5c0c28 P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193), 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:46:01.355965 4770 raft_consensus.cc:2454] T 6c9adf5004a649979b9abb97df5c0c28 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 0.
I20240502 20:46:01.356351 4637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c9adf5004a649979b9abb97df5c0c28" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" is_pre_election: true
I20240502 20:46:01.356468 4637 raft_consensus.cc:2454] T 6c9adf5004a649979b9abb97df5c0c28 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 0.
I20240502 20:46:01.356767 4441 leader_election.cc:302] T 6c9adf5004a649979b9abb97df5c0c28 P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, c3d26eafd51547acb780d4aac5ebe7cc; no voters:
I20240502 20:46:01.356995 6238 raft_consensus.cc:2781] T 6c9adf5004a649979b9abb97df5c0c28 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:01.356946 6244 raft_consensus.cc:504] T 04c72d7d63574dea99cb2f7c14199a73 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:01.357053 6238 raft_consensus.cc:482] T 6c9adf5004a649979b9abb97df5c0c28 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:01.357095 6238 raft_consensus.cc:3037] T 6c9adf5004a649979b9abb97df5c0c28 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:01.357347 6244 leader_election.cc:288] T 04c72d7d63574dea99cb2f7c14199a73 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 election: Requested vote from peers c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097), ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:46:01.357636 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04c72d7d63574dea99cb2f7c14199a73" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd"
I20240502 20:46:01.357654 4508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04c72d7d63574dea99cb2f7c14199a73" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc"
I20240502 20:46:01.357750 4508 raft_consensus.cc:3037] T 04c72d7d63574dea99cb2f7c14199a73 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:01.357748 4770 raft_consensus.cc:3037] T 04c72d7d63574dea99cb2f7c14199a73 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:01.358114 6238 raft_consensus.cc:504] T 6c9adf5004a649979b9abb97df5c0c28 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:01.358450 6238 leader_election.cc:288] T 6c9adf5004a649979b9abb97df5c0c28 P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 election: Requested vote from peers ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193), 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:46:01.358620 4508 raft_consensus.cc:2454] T 04c72d7d63574dea99cb2f7c14199a73 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 1.
I20240502 20:46:01.358687 4770 raft_consensus.cc:2454] T 04c72d7d63574dea99cb2f7c14199a73 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 1.
I20240502 20:46:01.358829 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c9adf5004a649979b9abb97df5c0c28" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd"
I20240502 20:46:01.358861 4637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c9adf5004a649979b9abb97df5c0c28" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91"
I20240502 20:46:01.358928 4768 raft_consensus.cc:3037] T 6c9adf5004a649979b9abb97df5c0c28 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:01.358922 4312 leader_election.cc:302] T 04c72d7d63574dea99cb2f7c14199a73 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3d26eafd51547acb780d4aac5ebe7cc, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:46:01.358956 4637 raft_consensus.cc:3037] T 6c9adf5004a649979b9abb97df5c0c28 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:01.359089 6244 raft_consensus.cc:2781] T 04c72d7d63574dea99cb2f7c14199a73 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:01.359189 6244 raft_consensus.cc:686] T 04c72d7d63574dea99cb2f7c14199a73 P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEADER]: Becoming Leader. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Running, Role: LEADER
I20240502 20:46:01.359333 6244 consensus_queue.cc:237] T 04c72d7d63574dea99cb2f7c14199a73 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:01.359812 4768 raft_consensus.cc:2454] T 6c9adf5004a649979b9abb97df5c0c28 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 1.
I20240502 20:46:01.359812 4637 raft_consensus.cc:2454] T 6c9adf5004a649979b9abb97df5c0c28 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 1.
I20240502 20:46:01.360186 4441 leader_election.cc:302] T 6c9adf5004a649979b9abb97df5c0c28 P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, c3d26eafd51547acb780d4aac5ebe7cc; no voters:
I20240502 20:46:01.360386 6238 raft_consensus.cc:2781] T 6c9adf5004a649979b9abb97df5c0c28 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:01.360486 6238 raft_consensus.cc:686] T 6c9adf5004a649979b9abb97df5c0c28 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Becoming Leader. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Running, Role: LEADER
I20240502 20:46:01.360611 6238 consensus_queue.cc:237] T 6c9adf5004a649979b9abb97df5c0c28 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:01.362268 4230 catalog_manager.cc:5484] T 04c72d7d63574dea99cb2f7c14199a73 P c92e6e46782849479c8a0c6d82276fb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65). New cstate: current_term: 1 leader_uuid: "c92e6e46782849479c8a0c6d82276fb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:01.362687 4226 catalog_manager.cc:5484] T 6c9adf5004a649979b9abb97df5c0c28 P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: term changed from 0 to 1, leader changed from <none> to c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66). New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:01.370339 6241 raft_consensus.cc:482] T e5a4dc24914c4f37a14481b77c4fcdaa P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:01.370427 6241 raft_consensus.cc:504] T e5a4dc24914c4f37a14481b77c4fcdaa P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:01.370786 6241 leader_election.cc:288] T e5a4dc24914c4f37a14481b77c4fcdaa P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097), c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:46:01.371070 4508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a4dc24914c4f37a14481b77c4fcdaa" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" is_pre_election: true
I20240502 20:46:01.371095 4376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a4dc24914c4f37a14481b77c4fcdaa" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2" is_pre_election: true
I20240502 20:46:01.371167 4508 raft_consensus.cc:2454] T e5a4dc24914c4f37a14481b77c4fcdaa P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 0.
I20240502 20:46:01.371203 4376 raft_consensus.cc:2454] T e5a4dc24914c4f37a14481b77c4fcdaa P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 0.
I20240502 20:46:01.371455 4705 leader_election.cc:302] T e5a4dc24914c4f37a14481b77c4fcdaa P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3d26eafd51547acb780d4aac5ebe7cc, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:46:01.371608 6241 raft_consensus.cc:2781] T e5a4dc24914c4f37a14481b77c4fcdaa P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:01.371665 6241 raft_consensus.cc:482] T e5a4dc24914c4f37a14481b77c4fcdaa P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:01.371703 6241 raft_consensus.cc:3037] T e5a4dc24914c4f37a14481b77c4fcdaa P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:01.372381 6241 raft_consensus.cc:504] T e5a4dc24914c4f37a14481b77c4fcdaa P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:01.372681 6241 leader_election.cc:288] T e5a4dc24914c4f37a14481b77c4fcdaa P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 election: Requested vote from peers c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097), c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:46:01.372957 4508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a4dc24914c4f37a14481b77c4fcdaa" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc"
I20240502 20:46:01.373051 4376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a4dc24914c4f37a14481b77c4fcdaa" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2"
I20240502 20:46:01.373088 4508 raft_consensus.cc:3037] T e5a4dc24914c4f37a14481b77c4fcdaa P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:01.373155 4376 raft_consensus.cc:3037] T e5a4dc24914c4f37a14481b77c4fcdaa P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:01.373735 4508 raft_consensus.cc:2454] T e5a4dc24914c4f37a14481b77c4fcdaa P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 1.
I20240502 20:46:01.373876 4376 raft_consensus.cc:2454] T e5a4dc24914c4f37a14481b77c4fcdaa P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 1.
I20240502 20:46:01.374028 4705 leader_election.cc:302] T e5a4dc24914c4f37a14481b77c4fcdaa P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3d26eafd51547acb780d4aac5ebe7cc, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:46:01.374182 6241 raft_consensus.cc:2781] T e5a4dc24914c4f37a14481b77c4fcdaa P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:01.374280 6241 raft_consensus.cc:686] T e5a4dc24914c4f37a14481b77c4fcdaa P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Becoming Leader. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Running, Role: LEADER
I20240502 20:46:01.374406 6241 consensus_queue.cc:237] T e5a4dc24914c4f37a14481b77c4fcdaa P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:01.376217 4226 catalog_manager.cc:5484] T e5a4dc24914c4f37a14481b77c4fcdaa P ed47c023452a497192f35d2f617bf9bd reported cstate change: term changed from 0 to 1, leader changed from <none> to ed47c023452a497192f35d2f617bf9bd (127.30.32.68). New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: UNKNOWN } } }
W20240502 20:46:01.507710 6274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:01.508041 6274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:01.511317 6274 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_14'
I20240502 20:46:01.590582 4376 raft_consensus.cc:1261] T e5a4dc24914c4f37a14481b77c4fcdaa P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:01.590788 4508 raft_consensus.cc:1261] T e5a4dc24914c4f37a14481b77c4fcdaa P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:01.591198 6241 consensus_queue.cc:1035] T e5a4dc24914c4f37a14481b77c4fcdaa P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:01.591548 6241 consensus_queue.cc:1035] T e5a4dc24914c4f37a14481b77c4fcdaa P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:01.593657 4768 raft_consensus.cc:1261] T 6c9adf5004a649979b9abb97df5c0c28 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:01.594408 4637 raft_consensus.cc:1261] T 6c9adf5004a649979b9abb97df5c0c28 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:01.596619 6036 consensus_queue.cc:1035] T 6c9adf5004a649979b9abb97df5c0c28 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:01.597399 6238 consensus_queue.cc:1035] T 6c9adf5004a649979b9abb97df5c0c28 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:01.613399 6309 mvcc.cc:205] Tried to move back new op lower bound from 7023340591469780992 to 7023340590590836736. Current Snapshot: MvccSnapshot[applied={T|T < 7023340591469780992}]
I20240502 20:46:01.617651 4768 raft_consensus.cc:1261] T 04c72d7d63574dea99cb2f7c14199a73 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:01.618100 6138 consensus_queue.cc:1035] T 04c72d7d63574dea99cb2f7c14199a73 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:01.628804 4506 raft_consensus.cc:1261] T 04c72d7d63574dea99cb2f7c14199a73 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:01.630527 5915 consensus_queue.cc:1035] T 04c72d7d63574dea99cb2f7c14199a73 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 108.943 ms
time per row: 0.0544715 ms
I20240502 20:46:01.665096 4227 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41812:
name: "rebalance_test_table_15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:46:01.670926 4486 tablet_service.cc:1467] Processing CreateTablet for tablet 8e7e145762bf458a8374bfc503025a17 (DEFAULT_TABLE table=rebalance_test_table_15 [id=57f2e4fb3cca4e408bd33dac3f4b9d0c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:01.671030 4618 tablet_service.cc:1467] Processing CreateTablet for tablet 8e7e145762bf458a8374bfc503025a17 (DEFAULT_TABLE table=rebalance_test_table_15 [id=57f2e4fb3cca4e408bd33dac3f4b9d0c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:01.671280 4486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e7e145762bf458a8374bfc503025a17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:01.671280 4618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e7e145762bf458a8374bfc503025a17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:01.671253 4617 tablet_service.cc:1467] Processing CreateTablet for tablet e7e0539e60064d97a7eb0eacd9df065e (DEFAULT_TABLE table=rebalance_test_table_15 [id=57f2e4fb3cca4e408bd33dac3f4b9d0c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:01.671943 4619 tablet_service.cc:1467] Processing CreateTablet for tablet 73efeba3d3094ef387e4efb03cff9d4e (DEFAULT_TABLE table=rebalance_test_table_15 [id=57f2e4fb3cca4e408bd33dac3f4b9d0c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:01.672163 4619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73efeba3d3094ef387e4efb03cff9d4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:01.672083 4749 tablet_service.cc:1467] Processing CreateTablet for tablet e7e0539e60064d97a7eb0eacd9df065e (DEFAULT_TABLE table=rebalance_test_table_15 [id=57f2e4fb3cca4e408bd33dac3f4b9d0c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:01.672360 4749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7e0539e60064d97a7eb0eacd9df065e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:01.673739 4750 tablet_service.cc:1467] Processing CreateTablet for tablet 73efeba3d3094ef387e4efb03cff9d4e (DEFAULT_TABLE table=rebalance_test_table_15 [id=57f2e4fb3cca4e408bd33dac3f4b9d0c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:01.673983 4750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73efeba3d3094ef387e4efb03cff9d4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:01.673979 4355 tablet_service.cc:1467] Processing CreateTablet for tablet 8e7e145762bf458a8374bfc503025a17 (DEFAULT_TABLE table=rebalance_test_table_15 [id=57f2e4fb3cca4e408bd33dac3f4b9d0c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:01.674261 4355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e7e145762bf458a8374bfc503025a17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:01.674455 4357 tablet_service.cc:1467] Processing CreateTablet for tablet 73efeba3d3094ef387e4efb03cff9d4e (DEFAULT_TABLE table=rebalance_test_table_15 [id=57f2e4fb3cca4e408bd33dac3f4b9d0c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:01.674677 4357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73efeba3d3094ef387e4efb03cff9d4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:01.676075 6261 tablet_bootstrap.cc:492] T 8e7e145762bf458a8374bfc503025a17 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:46:01.676460 6257 tablet_bootstrap.cc:492] T 8e7e145762bf458a8374bfc503025a17 P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:46:01.677129 6261 tablet_bootstrap.cc:654] T 8e7e145762bf458a8374bfc503025a17 P 923cfab926e7443f8ffd7b21ad0a9d91: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:01.677218 6258 tablet_bootstrap.cc:492] T e7e0539e60064d97a7eb0eacd9df065e P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:46:01.677218 4617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7e0539e60064d97a7eb0eacd9df065e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:01.678085 6258 tablet_bootstrap.cc:654] T e7e0539e60064d97a7eb0eacd9df065e P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:01.678357 6261 tablet_bootstrap.cc:492] T 8e7e145762bf458a8374bfc503025a17 P 923cfab926e7443f8ffd7b21ad0a9d91: No bootstrap required, opened a new log
I20240502 20:46:01.678457 6261 ts_tablet_manager.cc:1367] T 8e7e145762bf458a8374bfc503025a17 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:01.678634 6259 tablet_bootstrap.cc:492] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:46:01.678968 6261 raft_consensus.cc:348] T 8e7e145762bf458a8374bfc503025a17 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:01.679104 6261 raft_consensus.cc:374] T 8e7e145762bf458a8374bfc503025a17 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:01.679138 6261 raft_consensus.cc:729] T 8e7e145762bf458a8374bfc503025a17 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: FOLLOWER
I20240502 20:46:01.679298 6261 consensus_queue.cc:260] T 8e7e145762bf458a8374bfc503025a17 P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:01.679494 6259 tablet_bootstrap.cc:654] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:01.679735 6261 ts_tablet_manager.cc:1398] T 8e7e145762bf458a8374bfc503025a17 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 20:46:01.680124 6261 tablet_bootstrap.cc:492] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:46:01.680797 6259 tablet_bootstrap.cc:492] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:46:01.680879 6259 ts_tablet_manager.cc:1367] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:01.681067 6261 tablet_bootstrap.cc:654] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:01.681486 6259 raft_consensus.cc:348] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:01.681608 6259 raft_consensus.cc:374] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:01.681651 6259 raft_consensus.cc:729] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:46:01.681800 6259 consensus_queue.cc:260] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:01.682107 6259 ts_tablet_manager.cc:1398] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:01.682189 6257 tablet_bootstrap.cc:654] T 8e7e145762bf458a8374bfc503025a17 P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:01.682296 6259 tablet_bootstrap.cc:492] T 73efeba3d3094ef387e4efb03cff9d4e P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:46:01.683043 6258 tablet_bootstrap.cc:492] T e7e0539e60064d97a7eb0eacd9df065e P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:46:01.683135 6258 ts_tablet_manager.cc:1367] T e7e0539e60064d97a7eb0eacd9df065e P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 20:46:01.683205 6259 tablet_bootstrap.cc:654] T 73efeba3d3094ef387e4efb03cff9d4e P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:01.683548 6257 tablet_bootstrap.cc:492] T 8e7e145762bf458a8374bfc503025a17 P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:46:01.683638 6257 ts_tablet_manager.cc:1367] T 8e7e145762bf458a8374bfc503025a17 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240502 20:46:01.683594 6258 raft_consensus.cc:348] T e7e0539e60064d97a7eb0eacd9df065e P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:01.683694 6258 raft_consensus.cc:374] T e7e0539e60064d97a7eb0eacd9df065e P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:01.683730 6258 raft_consensus.cc:729] T e7e0539e60064d97a7eb0eacd9df065e P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:46:01.683859 6258 consensus_queue.cc:260] T e7e0539e60064d97a7eb0eacd9df065e P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:01.684152 6258 ts_tablet_manager.cc:1398] T e7e0539e60064d97a7eb0eacd9df065e P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:01.684098 6257 raft_consensus.cc:348] T 8e7e145762bf458a8374bfc503025a17 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:01.684212 6257 raft_consensus.cc:374] T 8e7e145762bf458a8374bfc503025a17 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:01.684245 6257 raft_consensus.cc:729] T 8e7e145762bf458a8374bfc503025a17 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:46:01.684331 6258 tablet_bootstrap.cc:492] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:46:01.684393 6259 tablet_bootstrap.cc:492] T 73efeba3d3094ef387e4efb03cff9d4e P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:46:01.684381 6257 consensus_queue.cc:260] T 8e7e145762bf458a8374bfc503025a17 P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:01.684469 6259 ts_tablet_manager.cc:1367] T 73efeba3d3094ef387e4efb03cff9d4e P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:01.684654 6257 ts_tablet_manager.cc:1398] T 8e7e145762bf458a8374bfc503025a17 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:01.685101 6259 raft_consensus.cc:348] T 73efeba3d3094ef387e4efb03cff9d4e P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:01.685209 6259 raft_consensus.cc:374] T 73efeba3d3094ef387e4efb03cff9d4e P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:01.685241 6259 raft_consensus.cc:729] T 73efeba3d3094ef387e4efb03cff9d4e P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:46:01.685374 6259 consensus_queue.cc:260] T 73efeba3d3094ef387e4efb03cff9d4e P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:01.685657 6259 ts_tablet_manager.cc:1398] T 73efeba3d3094ef387e4efb03cff9d4e P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:01.688541 4486 tablet_service.cc:1467] Processing CreateTablet for tablet e7e0539e60064d97a7eb0eacd9df065e (DEFAULT_TABLE table=rebalance_test_table_15 [id=57f2e4fb3cca4e408bd33dac3f4b9d0c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:01.688828 4486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7e0539e60064d97a7eb0eacd9df065e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:01.690783 6257 tablet_bootstrap.cc:492] T e7e0539e60064d97a7eb0eacd9df065e P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:46:01.691208 6258 tablet_bootstrap.cc:654] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:01.691792 6257 tablet_bootstrap.cc:654] T e7e0539e60064d97a7eb0eacd9df065e P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:01.692530 6261 tablet_bootstrap.cc:492] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91: No bootstrap required, opened a new log
I20240502 20:46:01.692530 6258 tablet_bootstrap.cc:492] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:46:01.692642 6261 ts_tablet_manager.cc:1367] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.002s
I20240502 20:46:01.692643 6258 ts_tablet_manager.cc:1367] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 20:46:01.692842 6257 tablet_bootstrap.cc:492] T e7e0539e60064d97a7eb0eacd9df065e P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:46:01.692922 6257 ts_tablet_manager.cc:1367] T e7e0539e60064d97a7eb0eacd9df065e P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:01.693100 6258 raft_consensus.cc:348] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:01.693101 6261 raft_consensus.cc:348] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:01.693235 6261 raft_consensus.cc:374] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:01.693235 6258 raft_consensus.cc:374] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:01.693272 6261 raft_consensus.cc:729] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: FOLLOWER
I20240502 20:46:01.693272 6258 raft_consensus.cc:729] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:46:01.693395 6261 consensus_queue.cc:260] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:01.693334 6257 raft_consensus.cc:348] T e7e0539e60064d97a7eb0eacd9df065e P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:01.693395 6258 consensus_queue.cc:260] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:01.693471 6257 raft_consensus.cc:374] T e7e0539e60064d97a7eb0eacd9df065e P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:01.693499 6257 raft_consensus.cc:729] T e7e0539e60064d97a7eb0eacd9df065e P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:46:01.693608 6257 consensus_queue.cc:260] T e7e0539e60064d97a7eb0eacd9df065e P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:01.693681 6261 ts_tablet_manager.cc:1398] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:01.693678 6258 ts_tablet_manager.cc:1398] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:01.693859 6261 tablet_bootstrap.cc:492] T e7e0539e60064d97a7eb0eacd9df065e P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:46:01.693934 6257 ts_tablet_manager.cc:1398] T e7e0539e60064d97a7eb0eacd9df065e P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:01.694687 6261 tablet_bootstrap.cc:654] T e7e0539e60064d97a7eb0eacd9df065e P 923cfab926e7443f8ffd7b21ad0a9d91: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:01.695878 6261 tablet_bootstrap.cc:492] T e7e0539e60064d97a7eb0eacd9df065e P 923cfab926e7443f8ffd7b21ad0a9d91: No bootstrap required, opened a new log
I20240502 20:46:01.695966 6261 ts_tablet_manager.cc:1367] T e7e0539e60064d97a7eb0eacd9df065e P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 20:46:01.696409 6261 raft_consensus.cc:348] T e7e0539e60064d97a7eb0eacd9df065e P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:01.696512 6261 raft_consensus.cc:374] T e7e0539e60064d97a7eb0eacd9df065e P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:01.696552 6261 raft_consensus.cc:729] T e7e0539e60064d97a7eb0eacd9df065e P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: FOLLOWER
I20240502 20:46:01.696705 6261 consensus_queue.cc:260] T e7e0539e60064d97a7eb0eacd9df065e P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:01.697079 6261 ts_tablet_manager.cc:1398] T e7e0539e60064d97a7eb0eacd9df065e P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:01.794701 6255 raft_consensus.cc:482] T e7e0539e60064d97a7eb0eacd9df065e P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:01.794842 6255 raft_consensus.cc:504] T e7e0539e60064d97a7eb0eacd9df065e P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:01.795302 6255 leader_election.cc:288] T e7e0539e60064d97a7eb0eacd9df065e P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097), ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:46:01.795696 4508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e0539e60064d97a7eb0eacd9df065e" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" is_pre_election: true
I20240502 20:46:01.795723 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e0539e60064d97a7eb0eacd9df065e" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd" is_pre_election: true
I20240502 20:46:01.795850 4508 raft_consensus.cc:2454] T e7e0539e60064d97a7eb0eacd9df065e P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 0.
I20240502 20:46:01.795905 4768 raft_consensus.cc:2454] T e7e0539e60064d97a7eb0eacd9df065e P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 0.
I20240502 20:46:01.796144 4574 leader_election.cc:302] T e7e0539e60064d97a7eb0eacd9df065e P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, c3d26eafd51547acb780d4aac5ebe7cc; no voters:
I20240502 20:46:01.796319 6255 raft_consensus.cc:2781] T e7e0539e60064d97a7eb0eacd9df065e P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:01.796398 6255 raft_consensus.cc:482] T e7e0539e60064d97a7eb0eacd9df065e P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:01.796449 6255 raft_consensus.cc:3037] T e7e0539e60064d97a7eb0eacd9df065e P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:01.797535 6255 raft_consensus.cc:504] T e7e0539e60064d97a7eb0eacd9df065e P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:01.797881 6255 leader_election.cc:288] T e7e0539e60064d97a7eb0eacd9df065e P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 election: Requested vote from peers c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097), ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:46:01.798208 4508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e0539e60064d97a7eb0eacd9df065e" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc"
I20240502 20:46:01.798280 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e0539e60064d97a7eb0eacd9df065e" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd"
I20240502 20:46:01.798339 4508 raft_consensus.cc:3037] T e7e0539e60064d97a7eb0eacd9df065e P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:01.798401 4768 raft_consensus.cc:3037] T e7e0539e60064d97a7eb0eacd9df065e P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:01.799108 4508 raft_consensus.cc:2454] T e7e0539e60064d97a7eb0eacd9df065e P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 1.
I20240502 20:46:01.799185 4768 raft_consensus.cc:2454] T e7e0539e60064d97a7eb0eacd9df065e P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 1.
I20240502 20:46:01.799465 4575 leader_election.cc:302] T e7e0539e60064d97a7eb0eacd9df065e P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:46:01.799654 6195 raft_consensus.cc:2781] T e7e0539e60064d97a7eb0eacd9df065e P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:01.799767 6195 raft_consensus.cc:686] T e7e0539e60064d97a7eb0eacd9df065e P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Becoming Leader. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Running, Role: LEADER
I20240502 20:46:01.799922 6195 consensus_queue.cc:237] T e7e0539e60064d97a7eb0eacd9df065e P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:01.801946 4231 catalog_manager.cc:5484] T e7e0539e60064d97a7eb0eacd9df065e P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67). New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:01.849481 5915 raft_consensus.cc:482] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:01.849611 5915 raft_consensus.cc:504] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:01.850057 5915 leader_election.cc:288] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097), 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:46:01.850440 4508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e7e145762bf458a8374bfc503025a17" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" is_pre_election: true
I20240502 20:46:01.850466 4637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e7e145762bf458a8374bfc503025a17" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" is_pre_election: true
I20240502 20:46:01.850584 4637 raft_consensus.cc:2454] T 8e7e145762bf458a8374bfc503025a17 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 0.
I20240502 20:46:01.850582 4508 raft_consensus.cc:2454] T 8e7e145762bf458a8374bfc503025a17 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 0.
I20240502 20:46:01.850875 4312 leader_election.cc:302] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3d26eafd51547acb780d4aac5ebe7cc, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:46:01.851068 5915 raft_consensus.cc:2781] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:01.851130 5915 raft_consensus.cc:482] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:01.851163 5915 raft_consensus.cc:3037] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:01.852002 5915 raft_consensus.cc:504] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:01.852330 5915 leader_election.cc:288] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 election: Requested vote from peers c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097), 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:46:01.852649 4508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e7e145762bf458a8374bfc503025a17" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc"
I20240502 20:46:01.852770 4508 raft_consensus.cc:3037] T 8e7e145762bf458a8374bfc503025a17 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:01.852808 4637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e7e145762bf458a8374bfc503025a17" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91"
I20240502 20:46:01.852931 4637 raft_consensus.cc:3037] T 8e7e145762bf458a8374bfc503025a17 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:01.853786 4508 raft_consensus.cc:2454] T 8e7e145762bf458a8374bfc503025a17 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 1.
I20240502 20:46:01.853916 4637 raft_consensus.cc:2454] T 8e7e145762bf458a8374bfc503025a17 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 1.
I20240502 20:46:01.854045 4312 leader_election.cc:302] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3d26eafd51547acb780d4aac5ebe7cc, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:46:01.854212 5915 raft_consensus.cc:2781] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:01.854305 5915 raft_consensus.cc:686] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEADER]: Becoming Leader. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Running, Role: LEADER
I20240502 20:46:01.854472 5915 consensus_queue.cc:237] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:01.856331 4231 catalog_manager.cc:5484] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65). New cstate: current_term: 1 leader_uuid: "c92e6e46782849479c8a0c6d82276fb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: HEALTHY } } }
I20240502 20:46:01.856518 6255 raft_consensus.cc:482] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:01.856603 6255 raft_consensus.cc:504] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:01.857254 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73efeba3d3094ef387e4efb03cff9d4e" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd" is_pre_election: true
I20240502 20:46:01.857388 4768 raft_consensus.cc:2454] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 0.
I20240502 20:46:01.857856 4375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73efeba3d3094ef387e4efb03cff9d4e" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2" is_pre_election: true
I20240502 20:46:01.857969 4375 raft_consensus.cc:2454] T 73efeba3d3094ef387e4efb03cff9d4e P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 0.
I20240502 20:46:01.857666 4575 leader_election.cc:302] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:46:01.856920 6255 leader_election.cc:288] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193), c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:46:01.858366 6255 raft_consensus.cc:2781] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:01.858413 6255 raft_consensus.cc:482] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:01.858445 6255 raft_consensus.cc:3037] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:01.859617 6255 raft_consensus.cc:504] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:01.859936 6255 leader_election.cc:288] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 election: Requested vote from peers ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193), c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:46:01.860277 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73efeba3d3094ef387e4efb03cff9d4e" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd"
I20240502 20:46:01.860299 4375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73efeba3d3094ef387e4efb03cff9d4e" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2"
I20240502 20:46:01.860386 4768 raft_consensus.cc:3037] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:01.860399 4375 raft_consensus.cc:3037] T 73efeba3d3094ef387e4efb03cff9d4e P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:01.861296 4375 raft_consensus.cc:2454] T 73efeba3d3094ef387e4efb03cff9d4e P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 1.
I20240502 20:46:01.861410 4768 raft_consensus.cc:2454] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 1.
I20240502 20:46:01.861673 4575 leader_election.cc:302] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:46:01.861985 6255 raft_consensus.cc:2781] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:01.862455 6255 raft_consensus.cc:686] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Becoming Leader. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Running, Role: LEADER
I20240502 20:46:01.862591 6255 consensus_queue.cc:237] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:01.864362 4231 catalog_manager.cc:5484] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67). New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: UNKNOWN } } }
W20240502 20:46:01.984145 6331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:01.984387 6331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:01.987783 6331 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_15'
I20240502 20:46:02.079547 4375 raft_consensus.cc:1261] T 73efeba3d3094ef387e4efb03cff9d4e P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:02.079694 4768 raft_consensus.cc:1261] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:02.080713 6329 consensus_queue.cc:1035] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:02.080789 4508 raft_consensus.cc:1261] T e7e0539e60064d97a7eb0eacd9df065e P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:02.081032 6329 consensus_queue.cc:1035] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:02.081135 4637 raft_consensus.cc:1261] T 8e7e145762bf458a8374bfc503025a17 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:02.081507 5915 consensus_queue.cc:1035] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:02.081668 4768 raft_consensus.cc:1261] T e7e0539e60064d97a7eb0eacd9df065e P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:02.082142 6195 consensus_queue.cc:1035] T e7e0539e60064d97a7eb0eacd9df065e P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:02.082386 6195 consensus_queue.cc:1035] T e7e0539e60064d97a7eb0eacd9df065e P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:02.089267 4506 raft_consensus.cc:1261] T 8e7e145762bf458a8374bfc503025a17 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:02.093113 6138 consensus_queue.cc:1035] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:02.102937 6224 mvcc.cc:205] Tried to move back new op lower bound from 7023340593474011136 to 7023340592590213120. Current Snapshot: MvccSnapshot[applied={T|T < 7023340593474011136}]
INSERT report
rows total: 2000
time total: 115.395 ms
time per row: 0.0576975 ms
I20240502 20:46:02.148967 4227 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41812:
name: "rebalance_test_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:46:02.155473 4617 tablet_service.cc:1467] Processing CreateTablet for tablet 5a36bf6e8e6e43b2b47b6d6357b0d445 (DEFAULT_TABLE table=rebalance_test_table_16 [id=dffd2e92869942b9a6f03505945ad3ba]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:02.155828 4617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a36bf6e8e6e43b2b47b6d6357b0d445. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:02.155995 4488 tablet_service.cc:1467] Processing CreateTablet for tablet f09dcfea3eb347e687f3c72f4ccc6929 (DEFAULT_TABLE table=rebalance_test_table_16 [id=dffd2e92869942b9a6f03505945ad3ba]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:02.156025 4486 tablet_service.cc:1467] Processing CreateTablet for tablet 5a36bf6e8e6e43b2b47b6d6357b0d445 (DEFAULT_TABLE table=rebalance_test_table_16 [id=dffd2e92869942b9a6f03505945ad3ba]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:02.156025 4487 tablet_service.cc:1467] Processing CreateTablet for tablet 628c976198d64c26ba1d68492416772a (DEFAULT_TABLE table=rebalance_test_table_16 [id=dffd2e92869942b9a6f03505945ad3ba]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:02.156275 4488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f09dcfea3eb347e687f3c72f4ccc6929. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:02.156430 4487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 628c976198d64c26ba1d68492416772a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:02.157099 4355 tablet_service.cc:1467] Processing CreateTablet for tablet f09dcfea3eb347e687f3c72f4ccc6929 (DEFAULT_TABLE table=rebalance_test_table_16 [id=dffd2e92869942b9a6f03505945ad3ba]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:02.157372 4355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f09dcfea3eb347e687f3c72f4ccc6929. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:02.158368 4619 tablet_service.cc:1467] Processing CreateTablet for tablet 628c976198d64c26ba1d68492416772a (DEFAULT_TABLE table=rebalance_test_table_16 [id=dffd2e92869942b9a6f03505945ad3ba]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:02.158687 4619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 628c976198d64c26ba1d68492416772a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:02.159334 6259 tablet_bootstrap.cc:492] T f09dcfea3eb347e687f3c72f4ccc6929 P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:46:02.159387 6257 tablet_bootstrap.cc:492] T 628c976198d64c26ba1d68492416772a P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:46:02.160084 6257 tablet_bootstrap.cc:654] T 628c976198d64c26ba1d68492416772a P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:02.160337 6259 tablet_bootstrap.cc:654] T f09dcfea3eb347e687f3c72f4ccc6929 P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:02.160864 4357 tablet_service.cc:1467] Processing CreateTablet for tablet 5a36bf6e8e6e43b2b47b6d6357b0d445 (DEFAULT_TABLE table=rebalance_test_table_16 [id=dffd2e92869942b9a6f03505945ad3ba]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:02.161108 6261 tablet_bootstrap.cc:492] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:46:02.161087 4750 tablet_service.cc:1467] Processing CreateTablet for tablet f09dcfea3eb347e687f3c72f4ccc6929 (DEFAULT_TABLE table=rebalance_test_table_16 [id=dffd2e92869942b9a6f03505945ad3ba]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:02.161337 4750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f09dcfea3eb347e687f3c72f4ccc6929. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:02.161520 6257 tablet_bootstrap.cc:492] T 628c976198d64c26ba1d68492416772a P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:46:02.161602 6257 ts_tablet_manager.cc:1367] T 628c976198d64c26ba1d68492416772a P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:02.161763 6261 tablet_bootstrap.cc:654] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 923cfab926e7443f8ffd7b21ad0a9d91: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:02.161972 6257 raft_consensus.cc:348] T 628c976198d64c26ba1d68492416772a P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:02.162070 6257 raft_consensus.cc:374] T 628c976198d64c26ba1d68492416772a P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:02.162110 6257 raft_consensus.cc:729] T 628c976198d64c26ba1d68492416772a P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:46:02.162242 6257 consensus_queue.cc:260] T 628c976198d64c26ba1d68492416772a P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:02.162537 6257 ts_tablet_manager.cc:1398] T 628c976198d64c26ba1d68492416772a P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:02.162552 4357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a36bf6e8e6e43b2b47b6d6357b0d445. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:02.162724 6257 tablet_bootstrap.cc:492] T f09dcfea3eb347e687f3c72f4ccc6929 P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:46:02.163563 6257 tablet_bootstrap.cc:654] T f09dcfea3eb347e687f3c72f4ccc6929 P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:02.163727 4749 tablet_service.cc:1467] Processing CreateTablet for tablet 628c976198d64c26ba1d68492416772a (DEFAULT_TABLE table=rebalance_test_table_16 [id=dffd2e92869942b9a6f03505945ad3ba]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:02.163951 4749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 628c976198d64c26ba1d68492416772a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:02.164566 6258 tablet_bootstrap.cc:492] T f09dcfea3eb347e687f3c72f4ccc6929 P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:46:02.164597 6257 tablet_bootstrap.cc:492] T f09dcfea3eb347e687f3c72f4ccc6929 P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:46:02.164666 6257 ts_tablet_manager.cc:1367] T f09dcfea3eb347e687f3c72f4ccc6929 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:02.165136 6257 raft_consensus.cc:348] T f09dcfea3eb347e687f3c72f4ccc6929 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:02.165246 6257 raft_consensus.cc:374] T f09dcfea3eb347e687f3c72f4ccc6929 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:02.165290 6257 raft_consensus.cc:729] T f09dcfea3eb347e687f3c72f4ccc6929 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:46:02.165410 6258 tablet_bootstrap.cc:654] T f09dcfea3eb347e687f3c72f4ccc6929 P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:02.165441 6257 consensus_queue.cc:260] T f09dcfea3eb347e687f3c72f4ccc6929 P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:02.165760 4486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a36bf6e8e6e43b2b47b6d6357b0d445. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:02.165794 6257 ts_tablet_manager.cc:1398] T f09dcfea3eb347e687f3c72f4ccc6929 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:02.166383 6261 tablet_bootstrap.cc:492] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 923cfab926e7443f8ffd7b21ad0a9d91: No bootstrap required, opened a new log
I20240502 20:46:02.166469 6261 ts_tablet_manager.cc:1367] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240502 20:46:02.166879 6261 raft_consensus.cc:348] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:02.166970 6261 raft_consensus.cc:374] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:02.167003 6261 raft_consensus.cc:729] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: FOLLOWER
I20240502 20:46:02.167137 6261 consensus_queue.cc:260] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:02.167397 6261 ts_tablet_manager.cc:1398] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:02.167554 6261 tablet_bootstrap.cc:492] T 628c976198d64c26ba1d68492416772a P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:46:02.167553 6257 tablet_bootstrap.cc:492] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:46:02.168426 6261 tablet_bootstrap.cc:654] T 628c976198d64c26ba1d68492416772a P 923cfab926e7443f8ffd7b21ad0a9d91: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:02.168426 6257 tablet_bootstrap.cc:654] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:02.169517 6261 tablet_bootstrap.cc:492] T 628c976198d64c26ba1d68492416772a P 923cfab926e7443f8ffd7b21ad0a9d91: No bootstrap required, opened a new log
I20240502 20:46:02.169663 6261 ts_tablet_manager.cc:1367] T 628c976198d64c26ba1d68492416772a P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:02.170532 6261 raft_consensus.cc:348] T 628c976198d64c26ba1d68492416772a P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:02.170656 6258 tablet_bootstrap.cc:492] T f09dcfea3eb347e687f3c72f4ccc6929 P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:46:02.170745 6258 ts_tablet_manager.cc:1367] T f09dcfea3eb347e687f3c72f4ccc6929 P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240502 20:46:02.171119 6258 raft_consensus.cc:348] T f09dcfea3eb347e687f3c72f4ccc6929 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:02.171221 6258 raft_consensus.cc:374] T f09dcfea3eb347e687f3c72f4ccc6929 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:02.171260 6258 raft_consensus.cc:729] T f09dcfea3eb347e687f3c72f4ccc6929 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:46:02.171384 6261 raft_consensus.cc:374] T 628c976198d64c26ba1d68492416772a P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:02.171377 6258 consensus_queue.cc:260] T f09dcfea3eb347e687f3c72f4ccc6929 P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:02.171502 6261 raft_consensus.cc:729] T 628c976198d64c26ba1d68492416772a P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: FOLLOWER
I20240502 20:46:02.171662 6258 ts_tablet_manager.cc:1398] T f09dcfea3eb347e687f3c72f4ccc6929 P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:02.171778 6261 consensus_queue.cc:260] T 628c976198d64c26ba1d68492416772a P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:02.171837 6258 tablet_bootstrap.cc:492] T 628c976198d64c26ba1d68492416772a P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:46:02.172488 6261 ts_tablet_manager.cc:1398] T 628c976198d64c26ba1d68492416772a P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 20:46:02.172708 6258 tablet_bootstrap.cc:654] T 628c976198d64c26ba1d68492416772a P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:02.173406 6257 tablet_bootstrap.cc:492] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:46:02.173497 6257 ts_tablet_manager.cc:1367] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240502 20:46:02.173978 6257 raft_consensus.cc:348] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:02.174070 6257 raft_consensus.cc:374] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:02.174093 6257 raft_consensus.cc:729] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:46:02.174193 6257 consensus_queue.cc:260] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:02.174444 6257 ts_tablet_manager.cc:1398] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:02.175626 6259 tablet_bootstrap.cc:492] T f09dcfea3eb347e687f3c72f4ccc6929 P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:46:02.175724 6259 ts_tablet_manager.cc:1367] T f09dcfea3eb347e687f3c72f4ccc6929 P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240502 20:46:02.176142 6259 raft_consensus.cc:348] T f09dcfea3eb347e687f3c72f4ccc6929 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:02.176229 6259 raft_consensus.cc:374] T f09dcfea3eb347e687f3c72f4ccc6929 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:02.176250 6259 raft_consensus.cc:729] T f09dcfea3eb347e687f3c72f4ccc6929 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:46:02.176349 6259 consensus_queue.cc:260] T f09dcfea3eb347e687f3c72f4ccc6929 P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:02.176674 6259 ts_tablet_manager.cc:1398] T f09dcfea3eb347e687f3c72f4ccc6929 P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:02.176932 6259 tablet_bootstrap.cc:492] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:46:02.177240 6258 tablet_bootstrap.cc:492] T 628c976198d64c26ba1d68492416772a P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:46:02.177309 6258 ts_tablet_manager.cc:1367] T 628c976198d64c26ba1d68492416772a P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 20:46:02.177754 6258 raft_consensus.cc:348] T 628c976198d64c26ba1d68492416772a P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:02.177876 6258 raft_consensus.cc:374] T 628c976198d64c26ba1d68492416772a P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:02.177922 6259 tablet_bootstrap.cc:654] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:02.177933 6258 raft_consensus.cc:729] T 628c976198d64c26ba1d68492416772a P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:46:02.178229 6258 consensus_queue.cc:260] T 628c976198d64c26ba1d68492416772a P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:02.178645 6258 ts_tablet_manager.cc:1398] T 628c976198d64c26ba1d68492416772a P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:02.179312 6259 tablet_bootstrap.cc:492] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:46:02.179523 6259 ts_tablet_manager.cc:1367] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:46:02.179924 6259 raft_consensus.cc:348] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:02.180014 6259 raft_consensus.cc:374] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:02.180051 6259 raft_consensus.cc:729] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:46:02.180188 6259 consensus_queue.cc:260] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:02.180614 6259 ts_tablet_manager.cc:1398] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:02.206187 6238 raft_consensus.cc:482] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:02.206326 6238 raft_consensus.cc:504] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:02.206748 6238 leader_election.cc:288] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827), 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:46:02.207242 4637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a36bf6e8e6e43b2b47b6d6357b0d445" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" is_pre_election: true
I20240502 20:46:02.207393 4637 raft_consensus.cc:2454] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 0.
I20240502 20:46:02.207479 4377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a36bf6e8e6e43b2b47b6d6357b0d445" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2" is_pre_election: true
I20240502 20:46:02.207614 4377 raft_consensus.cc:2454] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 0.
I20240502 20:46:02.207926 4441 leader_election.cc:302] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, c3d26eafd51547acb780d4aac5ebe7cc; no voters:
I20240502 20:46:02.208097 6238 raft_consensus.cc:2781] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:02.208160 6238 raft_consensus.cc:482] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:02.208199 6238 raft_consensus.cc:3037] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:02.209201 6238 raft_consensus.cc:504] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:02.209508 6238 leader_election.cc:288] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 election: Requested vote from peers c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827), 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:46:02.209841 4637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a36bf6e8e6e43b2b47b6d6357b0d445" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91"
I20240502 20:46:02.209892 4377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a36bf6e8e6e43b2b47b6d6357b0d445" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2"
I20240502 20:46:02.209959 4637 raft_consensus.cc:3037] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:02.210003 4377 raft_consensus.cc:3037] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:02.211010 4637 raft_consensus.cc:2454] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 1.
I20240502 20:46:02.211019 4377 raft_consensus.cc:2454] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 1.
I20240502 20:46:02.211328 4444 leader_election.cc:302] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3d26eafd51547acb780d4aac5ebe7cc, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:46:02.211552 6238 raft_consensus.cc:2781] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:02.211684 6238 raft_consensus.cc:686] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Becoming Leader. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Running, Role: LEADER
I20240502 20:46:02.211839 6238 consensus_queue.cc:237] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:02.214289 4231 catalog_manager.cc:5484] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: term changed from 0 to 1, leader changed from <none> to c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66). New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:02.254611 6238 raft_consensus.cc:482] T 628c976198d64c26ba1d68492416772a P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:02.254727 6238 raft_consensus.cc:504] T 628c976198d64c26ba1d68492416772a P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:02.255111 6238 leader_election.cc:288] T 628c976198d64c26ba1d68492416772a P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193), 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:46:02.255435 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "628c976198d64c26ba1d68492416772a" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd" is_pre_election: true
I20240502 20:46:02.255474 4637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "628c976198d64c26ba1d68492416772a" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" is_pre_election: true
I20240502 20:46:02.255563 4768 raft_consensus.cc:2454] T 628c976198d64c26ba1d68492416772a P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 0.
I20240502 20:46:02.255595 4637 raft_consensus.cc:2454] T 628c976198d64c26ba1d68492416772a P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 0.
I20240502 20:46:02.255874 4444 leader_election.cc:302] T 628c976198d64c26ba1d68492416772a P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3d26eafd51547acb780d4aac5ebe7cc, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:46:02.256049 6238 raft_consensus.cc:2781] T 628c976198d64c26ba1d68492416772a P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:02.256103 6238 raft_consensus.cc:482] T 628c976198d64c26ba1d68492416772a P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:02.256129 6238 raft_consensus.cc:3037] T 628c976198d64c26ba1d68492416772a P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:02.257014 6238 raft_consensus.cc:504] T 628c976198d64c26ba1d68492416772a P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:02.257347 6238 leader_election.cc:288] T 628c976198d64c26ba1d68492416772a P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 election: Requested vote from peers ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193), 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:46:02.257623 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "628c976198d64c26ba1d68492416772a" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd"
I20240502 20:46:02.257675 4637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "628c976198d64c26ba1d68492416772a" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91"
I20240502 20:46:02.257740 4768 raft_consensus.cc:3037] T 628c976198d64c26ba1d68492416772a P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:02.257776 4637 raft_consensus.cc:3037] T 628c976198d64c26ba1d68492416772a P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:02.258495 4768 raft_consensus.cc:2454] T 628c976198d64c26ba1d68492416772a P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 1.
I20240502 20:46:02.258546 4637 raft_consensus.cc:2454] T 628c976198d64c26ba1d68492416772a P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 1.
I20240502 20:46:02.258845 4441 leader_election.cc:302] T 628c976198d64c26ba1d68492416772a P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, c3d26eafd51547acb780d4aac5ebe7cc; no voters:
I20240502 20:46:02.259086 6238 raft_consensus.cc:2781] T 628c976198d64c26ba1d68492416772a P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:02.259181 6238 raft_consensus.cc:686] T 628c976198d64c26ba1d68492416772a P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Becoming Leader. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Running, Role: LEADER
I20240502 20:46:02.259341 6238 consensus_queue.cc:237] T 628c976198d64c26ba1d68492416772a P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:02.261162 4231 catalog_manager.cc:5484] T 628c976198d64c26ba1d68492416772a P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: term changed from 0 to 1, leader changed from <none> to c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66). New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:02.320140 6238 raft_consensus.cc:482] T f09dcfea3eb347e687f3c72f4ccc6929 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:02.320274 6238 raft_consensus.cc:504] T f09dcfea3eb347e687f3c72f4ccc6929 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:02.320581 6238 leader_election.cc:288] T f09dcfea3eb347e687f3c72f4ccc6929 P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827), ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:46:02.320946 4377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f09dcfea3eb347e687f3c72f4ccc6929" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2" is_pre_election: true
I20240502 20:46:02.321015 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f09dcfea3eb347e687f3c72f4ccc6929" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd" is_pre_election: true
I20240502 20:46:02.321133 4768 raft_consensus.cc:2454] T f09dcfea3eb347e687f3c72f4ccc6929 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 0.
I20240502 20:46:02.321134 4377 raft_consensus.cc:2454] T f09dcfea3eb347e687f3c72f4ccc6929 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 0.
I20240502 20:46:02.321440 4444 leader_election.cc:302] T f09dcfea3eb347e687f3c72f4ccc6929 P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3d26eafd51547acb780d4aac5ebe7cc, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:46:02.321602 6238 raft_consensus.cc:2781] T f09dcfea3eb347e687f3c72f4ccc6929 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:02.321662 6238 raft_consensus.cc:482] T f09dcfea3eb347e687f3c72f4ccc6929 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:02.321712 6238 raft_consensus.cc:3037] T f09dcfea3eb347e687f3c72f4ccc6929 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:02.322569 6238 raft_consensus.cc:504] T f09dcfea3eb347e687f3c72f4ccc6929 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:02.322912 6238 leader_election.cc:288] T f09dcfea3eb347e687f3c72f4ccc6929 P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 election: Requested vote from peers c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827), ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:46:02.323256 4377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f09dcfea3eb347e687f3c72f4ccc6929" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2"
I20240502 20:46:02.323374 4377 raft_consensus.cc:3037] T f09dcfea3eb347e687f3c72f4ccc6929 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:02.323731 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f09dcfea3eb347e687f3c72f4ccc6929" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd"
I20240502 20:46:02.323824 4768 raft_consensus.cc:3037] T f09dcfea3eb347e687f3c72f4ccc6929 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:02.324168 4377 raft_consensus.cc:2454] T f09dcfea3eb347e687f3c72f4ccc6929 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 1.
I20240502 20:46:02.324453 4444 leader_election.cc:302] T f09dcfea3eb347e687f3c72f4ccc6929 P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3d26eafd51547acb780d4aac5ebe7cc, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:46:02.324584 4768 raft_consensus.cc:2454] T f09dcfea3eb347e687f3c72f4ccc6929 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 1.
I20240502 20:46:02.324601 6238 raft_consensus.cc:2781] T f09dcfea3eb347e687f3c72f4ccc6929 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:02.324672 6238 raft_consensus.cc:686] T f09dcfea3eb347e687f3c72f4ccc6929 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Becoming Leader. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Running, Role: LEADER
I20240502 20:46:02.324819 6238 consensus_queue.cc:237] T f09dcfea3eb347e687f3c72f4ccc6929 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:02.326581 4231 catalog_manager.cc:5484] T f09dcfea3eb347e687f3c72f4ccc6929 P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: term changed from 0 to 1, leader changed from <none> to c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66). New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } }
W20240502 20:46:02.470772 6385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:02.471032 6385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:02.474145 6385 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_16'
I20240502 20:46:02.559482 4637 raft_consensus.cc:1261] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:02.559535 4768 raft_consensus.cc:1261] T 628c976198d64c26ba1d68492416772a P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:02.559893 4377 raft_consensus.cc:1261] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:02.560110 6375 consensus_queue.cc:1035] T 628c976198d64c26ba1d68492416772a P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:02.561563 6375 consensus_queue.cc:1035] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:02.563819 6238 consensus_queue.cc:1035] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:02.564800 4637 raft_consensus.cc:1261] T 628c976198d64c26ba1d68492416772a P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:02.565690 4376 raft_consensus.cc:1261] T f09dcfea3eb347e687f3c72f4ccc6929 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:02.565923 4770 raft_consensus.cc:1261] T f09dcfea3eb347e687f3c72f4ccc6929 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:02.566372 6238 consensus_queue.cc:1035] T f09dcfea3eb347e687f3c72f4ccc6929 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:02.566524 6375 consensus_queue.cc:1035] T f09dcfea3eb347e687f3c72f4ccc6929 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:02.574402 6238 consensus_queue.cc:1035] T 628c976198d64c26ba1d68492416772a P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 112.521 ms
time per row: 0.0562604 ms
I20240502 20:46:02.622869 6361 mvcc.cc:205] Tried to move back new op lower bound from 7023340595454488576 to 7023340594215510016. Current Snapshot: MvccSnapshot[applied={T|T < 7023340595454488576 or (T in {7023340595454488576})}]
I20240502 20:46:02.630432 4230 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41812:
name: "rebalance_test_table_17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:46:02.635783 4357 tablet_service.cc:1467] Processing CreateTablet for tablet 0cf7e95cb91b4e74b14365fcc6b7f968 (DEFAULT_TABLE table=rebalance_test_table_17 [id=75dfa602f9634de99d6f55e4fdeea21c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:02.636065 4357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cf7e95cb91b4e74b14365fcc6b7f968. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:02.636729 4486 tablet_service.cc:1467] Processing CreateTablet for tablet a62a7ce526f14c25994bb9e0e3c3bc84 (DEFAULT_TABLE table=rebalance_test_table_17 [id=75dfa602f9634de99d6f55e4fdeea21c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:02.636996 4486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a62a7ce526f14c25994bb9e0e3c3bc84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:02.637468 4619 tablet_service.cc:1467] Processing CreateTablet for tablet 0cf7e95cb91b4e74b14365fcc6b7f968 (DEFAULT_TABLE table=rebalance_test_table_17 [id=75dfa602f9634de99d6f55e4fdeea21c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:02.637543 4617 tablet_service.cc:1467] Processing CreateTablet for tablet a7ea68e8581a4cd5a6212abc96dfbcde (DEFAULT_TABLE table=rebalance_test_table_17 [id=75dfa602f9634de99d6f55e4fdeea21c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:02.637714 4619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cf7e95cb91b4e74b14365fcc6b7f968. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:02.637919 4617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7ea68e8581a4cd5a6212abc96dfbcde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:02.638123 4618 tablet_service.cc:1467] Processing CreateTablet for tablet a62a7ce526f14c25994bb9e0e3c3bc84 (DEFAULT_TABLE table=rebalance_test_table_17 [id=75dfa602f9634de99d6f55e4fdeea21c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:02.638335 4618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a62a7ce526f14c25994bb9e0e3c3bc84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:02.639014 6257 tablet_bootstrap.cc:492] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:46:02.639820 6259 tablet_bootstrap.cc:492] T 0cf7e95cb91b4e74b14365fcc6b7f968 P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:46:02.639923 6257 tablet_bootstrap.cc:654] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:02.640637 6259 tablet_bootstrap.cc:654] T 0cf7e95cb91b4e74b14365fcc6b7f968 P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:02.641173 6261 tablet_bootstrap.cc:492] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:46:02.641521 6259 tablet_bootstrap.cc:492] T 0cf7e95cb91b4e74b14365fcc6b7f968 P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:46:02.641602 6259 ts_tablet_manager.cc:1367] T 0cf7e95cb91b4e74b14365fcc6b7f968 P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:02.641953 6261 tablet_bootstrap.cc:654] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:02.642078 6259 raft_consensus.cc:348] T 0cf7e95cb91b4e74b14365fcc6b7f968 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:02.642207 6259 raft_consensus.cc:374] T 0cf7e95cb91b4e74b14365fcc6b7f968 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:02.642246 6259 raft_consensus.cc:729] T 0cf7e95cb91b4e74b14365fcc6b7f968 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:46:02.642382 6259 consensus_queue.cc:260] T 0cf7e95cb91b4e74b14365fcc6b7f968 P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:02.642697 6259 ts_tablet_manager.cc:1398] T 0cf7e95cb91b4e74b14365fcc6b7f968 P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:02.642771 6257 tablet_bootstrap.cc:492] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:46:02.642848 6257 ts_tablet_manager.cc:1367] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240502 20:46:02.643285 6257 raft_consensus.cc:348] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:02.643400 6257 raft_consensus.cc:374] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:02.643435 6257 raft_consensus.cc:729] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:46:02.643575 6257 consensus_queue.cc:260] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:02.643885 6257 ts_tablet_manager.cc:1398] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:02.644758 4749 tablet_service.cc:1467] Processing CreateTablet for tablet 0cf7e95cb91b4e74b14365fcc6b7f968 (DEFAULT_TABLE table=rebalance_test_table_17 [id=75dfa602f9634de99d6f55e4fdeea21c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:02.645009 4749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cf7e95cb91b4e74b14365fcc6b7f968. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:02.647342 4750 tablet_service.cc:1467] Processing CreateTablet for tablet a7ea68e8581a4cd5a6212abc96dfbcde (DEFAULT_TABLE table=rebalance_test_table_17 [id=75dfa602f9634de99d6f55e4fdeea21c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:02.647558 4750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7ea68e8581a4cd5a6212abc96dfbcde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:02.649152 6258 tablet_bootstrap.cc:492] T 0cf7e95cb91b4e74b14365fcc6b7f968 P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:46:02.649979 6258 tablet_bootstrap.cc:654] T 0cf7e95cb91b4e74b14365fcc6b7f968 P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:02.651016 6258 tablet_bootstrap.cc:492] T 0cf7e95cb91b4e74b14365fcc6b7f968 P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:46:02.651031 4355 tablet_service.cc:1467] Processing CreateTablet for tablet a62a7ce526f14c25994bb9e0e3c3bc84 (DEFAULT_TABLE table=rebalance_test_table_17 [id=75dfa602f9634de99d6f55e4fdeea21c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:02.651103 6258 ts_tablet_manager.cc:1367] T 0cf7e95cb91b4e74b14365fcc6b7f968 P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:02.651293 4355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a62a7ce526f14c25994bb9e0e3c3bc84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:02.651403 6261 tablet_bootstrap.cc:492] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91: No bootstrap required, opened a new log
I20240502 20:46:02.651414 4357 tablet_service.cc:1467] Processing CreateTablet for tablet a7ea68e8581a4cd5a6212abc96dfbcde (DEFAULT_TABLE table=rebalance_test_table_17 [id=75dfa602f9634de99d6f55e4fdeea21c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:02.651485 6261 ts_tablet_manager.cc:1367] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240502 20:46:02.651552 6258 raft_consensus.cc:348] T 0cf7e95cb91b4e74b14365fcc6b7f968 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:02.651630 4357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7ea68e8581a4cd5a6212abc96dfbcde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:02.651654 6258 raft_consensus.cc:374] T 0cf7e95cb91b4e74b14365fcc6b7f968 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:02.651692 6258 raft_consensus.cc:729] T 0cf7e95cb91b4e74b14365fcc6b7f968 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:46:02.651803 6258 consensus_queue.cc:260] T 0cf7e95cb91b4e74b14365fcc6b7f968 P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:02.651891 6261 raft_consensus.cc:348] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:02.651993 6261 raft_consensus.cc:374] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:02.652038 6261 raft_consensus.cc:729] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: FOLLOWER
I20240502 20:46:02.652101 6258 ts_tablet_manager.cc:1398] T 0cf7e95cb91b4e74b14365fcc6b7f968 P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:02.652159 6261 consensus_queue.cc:260] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:02.652254 6258 tablet_bootstrap.cc:492] T a7ea68e8581a4cd5a6212abc96dfbcde P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:46:02.652472 6261 ts_tablet_manager.cc:1398] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:02.652688 6261 tablet_bootstrap.cc:492] T a62a7ce526f14c25994bb9e0e3c3bc84 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:46:02.653085 6369 raft_consensus.cc:482] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:02.653184 6369 raft_consensus.cc:504] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:02.653594 6369 leader_election.cc:288] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827), ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:46:02.653928 6258 tablet_bootstrap.cc:654] T a7ea68e8581a4cd5a6212abc96dfbcde P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:02.654278 6261 tablet_bootstrap.cc:654] T a62a7ce526f14c25994bb9e0e3c3bc84 P 923cfab926e7443f8ffd7b21ad0a9d91: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:02.654984 6258 tablet_bootstrap.cc:492] T a7ea68e8581a4cd5a6212abc96dfbcde P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:46:02.655055 6258 ts_tablet_manager.cc:1367] T a7ea68e8581a4cd5a6212abc96dfbcde P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:46:02.655417 6258 raft_consensus.cc:348] T a7ea68e8581a4cd5a6212abc96dfbcde P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:02.655529 6258 raft_consensus.cc:374] T a7ea68e8581a4cd5a6212abc96dfbcde P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:02.655551 6259 tablet_bootstrap.cc:492] T a62a7ce526f14c25994bb9e0e3c3bc84 P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:46:02.655561 6258 raft_consensus.cc:729] T a7ea68e8581a4cd5a6212abc96dfbcde P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:46:02.655664 6258 consensus_queue.cc:260] T a7ea68e8581a4cd5a6212abc96dfbcde P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:02.655858 6261 tablet_bootstrap.cc:492] T a62a7ce526f14c25994bb9e0e3c3bc84 P 923cfab926e7443f8ffd7b21ad0a9d91: No bootstrap required, opened a new log
I20240502 20:46:02.655933 6261 ts_tablet_manager.cc:1367] T a62a7ce526f14c25994bb9e0e3c3bc84 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:46:02.655931 6258 ts_tablet_manager.cc:1398] T a7ea68e8581a4cd5a6212abc96dfbcde P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:02.656405 6259 tablet_bootstrap.cc:654] T a62a7ce526f14c25994bb9e0e3c3bc84 P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:02.656343 6261 raft_consensus.cc:348] T a62a7ce526f14c25994bb9e0e3c3bc84 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:02.656450 6261 raft_consensus.cc:374] T a62a7ce526f14c25994bb9e0e3c3bc84 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:02.656478 6261 raft_consensus.cc:729] T a62a7ce526f14c25994bb9e0e3c3bc84 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: FOLLOWER
I20240502 20:46:02.656603 6261 consensus_queue.cc:260] T a62a7ce526f14c25994bb9e0e3c3bc84 P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:02.656886 6261 ts_tablet_manager.cc:1398] T a62a7ce526f14c25994bb9e0e3c3bc84 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:02.656875 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ea68e8581a4cd5a6212abc96dfbcde" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd" is_pre_election: true
I20240502 20:46:02.657008 4768 raft_consensus.cc:2454] T a7ea68e8581a4cd5a6212abc96dfbcde P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 0.
I20240502 20:46:02.657313 6261 tablet_bootstrap.cc:492] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:46:02.658176 6261 tablet_bootstrap.cc:654] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91: Neither blocks nor log segments found. Creating new log.
W20240502 20:46:02.658131 4575 leader_election.cc:341] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827): Not found: Tablet not found: a7ea68e8581a4cd5a6212abc96dfbcde
I20240502 20:46:02.659044 4575 leader_election.cc:302] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, ed47c023452a497192f35d2f617bf9bd; no voters: c92e6e46782849479c8a0c6d82276fb2
I20240502 20:46:02.659209 6368 raft_consensus.cc:2781] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:02.659276 6368 raft_consensus.cc:482] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:02.659322 6368 raft_consensus.cc:3037] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:02.660377 6259 tablet_bootstrap.cc:492] T a62a7ce526f14c25994bb9e0e3c3bc84 P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:46:02.660384 6368 raft_consensus.cc:504] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:02.660465 6259 ts_tablet_manager.cc:1367] T a62a7ce526f14c25994bb9e0e3c3bc84 P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 20:46:02.660710 6368 leader_election.cc:288] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 election: Requested vote from peers c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827), ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:46:02.660908 6259 raft_consensus.cc:348] T a62a7ce526f14c25994bb9e0e3c3bc84 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:02.661026 6259 raft_consensus.cc:374] T a62a7ce526f14c25994bb9e0e3c3bc84 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:02.661096 6259 raft_consensus.cc:729] T a62a7ce526f14c25994bb9e0e3c3bc84 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:46:02.661213 6259 consensus_queue.cc:260] T a62a7ce526f14c25994bb9e0e3c3bc84 P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:02.661509 6259 ts_tablet_manager.cc:1398] T a62a7ce526f14c25994bb9e0e3c3bc84 P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:02.662102 6259 tablet_bootstrap.cc:492] T a7ea68e8581a4cd5a6212abc96dfbcde P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:46:02.662166 4376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ea68e8581a4cd5a6212abc96dfbcde" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2"
I20240502 20:46:02.662184 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ea68e8581a4cd5a6212abc96dfbcde" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd"
I20240502 20:46:02.662276 4768 raft_consensus.cc:3037] T a7ea68e8581a4cd5a6212abc96dfbcde P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
W20240502 20:46:02.662513 4575 leader_election.cc:341] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827): Illegal state: must be running to vote when last-logged opid is not known
I20240502 20:46:02.662868 6259 tablet_bootstrap.cc:654] T a7ea68e8581a4cd5a6212abc96dfbcde P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:02.663051 4768 raft_consensus.cc:2454] T a7ea68e8581a4cd5a6212abc96dfbcde P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 1.
I20240502 20:46:02.663321 4575 leader_election.cc:302] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, ed47c023452a497192f35d2f617bf9bd; no voters: c92e6e46782849479c8a0c6d82276fb2
I20240502 20:46:02.663532 6368 raft_consensus.cc:2781] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:02.663621 6368 raft_consensus.cc:686] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Becoming Leader. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Running, Role: LEADER
I20240502 20:46:02.663748 6368 consensus_queue.cc:237] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:02.664688 6261 tablet_bootstrap.cc:492] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91: No bootstrap required, opened a new log
I20240502 20:46:02.664768 6261 ts_tablet_manager.cc:1367] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240502 20:46:02.665419 6261 raft_consensus.cc:348] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:02.665526 6261 raft_consensus.cc:374] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:02.665570 6261 raft_consensus.cc:729] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: FOLLOWER
I20240502 20:46:02.665647 6259 tablet_bootstrap.cc:492] T a7ea68e8581a4cd5a6212abc96dfbcde P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:46:02.665728 6259 ts_tablet_manager.cc:1367] T a7ea68e8581a4cd5a6212abc96dfbcde P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 20:46:02.665709 6261 consensus_queue.cc:260] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:02.665979 6261 ts_tablet_manager.cc:1398] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:02.666072 6259 raft_consensus.cc:348] T a7ea68e8581a4cd5a6212abc96dfbcde P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:02.666182 6259 raft_consensus.cc:374] T a7ea68e8581a4cd5a6212abc96dfbcde P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:02.666215 6259 raft_consensus.cc:729] T a7ea68e8581a4cd5a6212abc96dfbcde P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:46:02.666344 6259 consensus_queue.cc:260] T a7ea68e8581a4cd5a6212abc96dfbcde P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:02.666589 6259 ts_tablet_manager.cc:1398] T a7ea68e8581a4cd5a6212abc96dfbcde P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:02.667083 4228 catalog_manager.cc:5484] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67). New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:02.726768 6368 raft_consensus.cc:482] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:02.726889 6368 raft_consensus.cc:504] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:02.727253 6368 leader_election.cc:288] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827), ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:46:02.727602 4376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cf7e95cb91b4e74b14365fcc6b7f968" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2" is_pre_election: true
I20240502 20:46:02.727689 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cf7e95cb91b4e74b14365fcc6b7f968" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd" is_pre_election: true
I20240502 20:46:02.727774 4376 raft_consensus.cc:2454] T 0cf7e95cb91b4e74b14365fcc6b7f968 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 0.
I20240502 20:46:02.727854 4768 raft_consensus.cc:2454] T 0cf7e95cb91b4e74b14365fcc6b7f968 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 0.
I20240502 20:46:02.728101 4575 leader_election.cc:302] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:46:02.728322 6368 raft_consensus.cc:2781] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:02.728384 6368 raft_consensus.cc:482] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:02.728431 6368 raft_consensus.cc:3037] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:02.729486 6368 raft_consensus.cc:504] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:02.729838 6368 leader_election.cc:288] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 election: Requested vote from peers c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827), ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:46:02.730160 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cf7e95cb91b4e74b14365fcc6b7f968" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd"
I20240502 20:46:02.730195 4376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cf7e95cb91b4e74b14365fcc6b7f968" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2"
I20240502 20:46:02.730315 4768 raft_consensus.cc:3037] T 0cf7e95cb91b4e74b14365fcc6b7f968 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:02.730334 4376 raft_consensus.cc:3037] T 0cf7e95cb91b4e74b14365fcc6b7f968 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:02.731130 4768 raft_consensus.cc:2454] T 0cf7e95cb91b4e74b14365fcc6b7f968 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 1.
I20240502 20:46:02.731225 4376 raft_consensus.cc:2454] T 0cf7e95cb91b4e74b14365fcc6b7f968 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 1.
I20240502 20:46:02.731415 4575 leader_election.cc:302] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:46:02.731663 6368 raft_consensus.cc:2781] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:02.731750 6368 raft_consensus.cc:686] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Becoming Leader. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Running, Role: LEADER
I20240502 20:46:02.732079 6368 consensus_queue.cc:237] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:02.734266 4228 catalog_manager.cc:5484] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67). New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:02.840966 6414 raft_consensus.cc:482] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:02.841150 6414 raft_consensus.cc:504] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:02.841609 6414 leader_election.cc:288] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827), 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:46:02.842000 4376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a62a7ce526f14c25994bb9e0e3c3bc84" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2" is_pre_election: true
I20240502 20:46:02.842082 4638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a62a7ce526f14c25994bb9e0e3c3bc84" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" is_pre_election: true
I20240502 20:46:02.842156 4376 raft_consensus.cc:2454] T a62a7ce526f14c25994bb9e0e3c3bc84 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 0.
I20240502 20:46:02.842206 4638 raft_consensus.cc:2454] T a62a7ce526f14c25994bb9e0e3c3bc84 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 0.
I20240502 20:46:02.842521 4444 leader_election.cc:302] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3d26eafd51547acb780d4aac5ebe7cc, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:46:02.842705 6414 raft_consensus.cc:2781] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:02.842777 6414 raft_consensus.cc:482] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:02.842826 6414 raft_consensus.cc:3037] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:02.843858 6414 raft_consensus.cc:504] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:02.844208 6414 leader_election.cc:288] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 election: Requested vote from peers c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827), 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:46:02.844554 4638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a62a7ce526f14c25994bb9e0e3c3bc84" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91"
I20240502 20:46:02.844712 4638 raft_consensus.cc:3037] T a62a7ce526f14c25994bb9e0e3c3bc84 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:02.845705 4638 raft_consensus.cc:2454] T a62a7ce526f14c25994bb9e0e3c3bc84 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 1.
I20240502 20:46:02.845952 4376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a62a7ce526f14c25994bb9e0e3c3bc84" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2"
I20240502 20:46:02.845990 4441 leader_election.cc:302] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, c3d26eafd51547acb780d4aac5ebe7cc; no voters:
I20240502 20:46:02.846071 4376 raft_consensus.cc:3037] T a62a7ce526f14c25994bb9e0e3c3bc84 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:02.846995 4376 raft_consensus.cc:2454] T a62a7ce526f14c25994bb9e0e3c3bc84 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 1.
I20240502 20:46:02.847200 6414 raft_consensus.cc:2781] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:02.847337 6414 raft_consensus.cc:686] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Becoming Leader. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Running, Role: LEADER
I20240502 20:46:02.847503 6414 consensus_queue.cc:237] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:02.849678 4228 catalog_manager.cc:5484] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: term changed from 0 to 1, leader changed from <none> to c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66). New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } }
I20240502 20:46:02.889554 6138 raft_consensus.cc:482] T a7ea68e8581a4cd5a6212abc96dfbcde P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:02.889722 6138 raft_consensus.cc:504] T a7ea68e8581a4cd5a6212abc96dfbcde P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:02.890340 6138 leader_election.cc:288] T a7ea68e8581a4cd5a6212abc96dfbcde P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141), ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:46:02.890755 4638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ea68e8581a4cd5a6212abc96dfbcde" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" is_pre_election: true
I20240502 20:46:02.890843 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ea68e8581a4cd5a6212abc96dfbcde" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd" is_pre_election: true
I20240502 20:46:02.891026 4768 raft_consensus.cc:2379] T a7ea68e8581a4cd5a6212abc96dfbcde P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c92e6e46782849479c8a0c6d82276fb2 in current term 1: Already voted for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in this term.
I20240502 20:46:02.891376 4313 leader_election.cc:302] T a7ea68e8581a4cd5a6212abc96dfbcde P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2; no voters: 923cfab926e7443f8ffd7b21ad0a9d91, ed47c023452a497192f35d2f617bf9bd
I20240502 20:46:02.891615 6138 raft_consensus.cc:3037] T a7ea68e8581a4cd5a6212abc96dfbcde P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:02.892639 6138 raft_consensus.cc:2726] T a7ea68e8581a4cd5a6212abc96dfbcde P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 20:46:02.998899 6436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:02.999176 6436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:03.002583 6436 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_17'
I20240502 20:46:03.087718 4637 raft_consensus.cc:1261] T a62a7ce526f14c25994bb9e0e3c3bc84 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:03.088165 6238 consensus_queue.cc:1035] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:03.089473 4376 raft_consensus.cc:1261] T a62a7ce526f14c25994bb9e0e3c3bc84 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:03.090001 4376 raft_consensus.cc:1261] T a7ea68e8581a4cd5a6212abc96dfbcde P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:03.090034 4768 raft_consensus.cc:1261] T a7ea68e8581a4cd5a6212abc96dfbcde P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:03.090451 6238 consensus_queue.cc:1035] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:03.090571 6369 consensus_queue.cc:1035] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:03.090900 6369 consensus_queue.cc:1035] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:03.100133 4770 raft_consensus.cc:1261] T 0cf7e95cb91b4e74b14365fcc6b7f968 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:03.100538 4374 raft_consensus.cc:1261] T 0cf7e95cb91b4e74b14365fcc6b7f968 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:03.101022 6368 consensus_queue.cc:1035] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:03.101166 6369 consensus_queue.cc:1035] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:03.111292 6411 mvcc.cc:205] Tried to move back new op lower bound from 7023340597602795520 to 7023340596625027072. Current Snapshot: MvccSnapshot[applied={T|T < 7023340597602795520 or (T in {7023340597602795520})}]
INSERT report
rows total: 2000
time total: 121.566 ms
time per row: 0.0607829 ms
I20240502 20:46:03.171571 4230 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41812:
name: "rebalance_test_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:46:03.177445 4750 tablet_service.cc:1467] Processing CreateTablet for tablet 66b8f583dfd24c2aa5a59457a8e58fbb (DEFAULT_TABLE table=rebalance_test_table_18 [id=393c8bbf6b174a9e891d96bcd09fe045]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:03.177601 4357 tablet_service.cc:1467] Processing CreateTablet for tablet 66b8f583dfd24c2aa5a59457a8e58fbb (DEFAULT_TABLE table=rebalance_test_table_18 [id=393c8bbf6b174a9e891d96bcd09fe045]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:03.177798 4750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66b8f583dfd24c2aa5a59457a8e58fbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:03.177862 4357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66b8f583dfd24c2aa5a59457a8e58fbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:03.180106 6483 tablet_bootstrap.cc:492] T 66b8f583dfd24c2aa5a59457a8e58fbb P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:46:03.180848 4619 tablet_service.cc:1467] Processing CreateTablet for tablet c6ac59f330454d2a89ad9c11192ed4f0 (DEFAULT_TABLE table=rebalance_test_table_18 [id=393c8bbf6b174a9e891d96bcd09fe045]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:03.181161 4619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6ac59f330454d2a89ad9c11192ed4f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:03.181538 4357 tablet_service.cc:1467] Processing CreateTablet for tablet a55d5b8b9ed04ca39888b89b82a3f833 (DEFAULT_TABLE table=rebalance_test_table_18 [id=393c8bbf6b174a9e891d96bcd09fe045]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:03.181540 4486 tablet_service.cc:1467] Processing CreateTablet for tablet 66b8f583dfd24c2aa5a59457a8e58fbb (DEFAULT_TABLE table=rebalance_test_table_18 [id=393c8bbf6b174a9e891d96bcd09fe045]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:03.181771 6483 tablet_bootstrap.cc:654] T 66b8f583dfd24c2aa5a59457a8e58fbb P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:03.181802 4486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66b8f583dfd24c2aa5a59457a8e58fbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:03.181779 4357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a55d5b8b9ed04ca39888b89b82a3f833. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:03.182585 6482 tablet_bootstrap.cc:492] T 66b8f583dfd24c2aa5a59457a8e58fbb P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:46:03.183658 4488 tablet_service.cc:1467] Processing CreateTablet for tablet c6ac59f330454d2a89ad9c11192ed4f0 (DEFAULT_TABLE table=rebalance_test_table_18 [id=393c8bbf6b174a9e891d96bcd09fe045]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:03.183774 6482 tablet_bootstrap.cc:654] T 66b8f583dfd24c2aa5a59457a8e58fbb P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:03.183931 4488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6ac59f330454d2a89ad9c11192ed4f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:03.184113 4487 tablet_service.cc:1467] Processing CreateTablet for tablet a55d5b8b9ed04ca39888b89b82a3f833 (DEFAULT_TABLE table=rebalance_test_table_18 [id=393c8bbf6b174a9e891d96bcd09fe045]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:03.184285 4487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a55d5b8b9ed04ca39888b89b82a3f833. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:03.184454 6483 tablet_bootstrap.cc:492] T 66b8f583dfd24c2aa5a59457a8e58fbb P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:46:03.184535 6483 ts_tablet_manager.cc:1367] T 66b8f583dfd24c2aa5a59457a8e58fbb P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 20:46:03.185065 6483 raft_consensus.cc:348] T 66b8f583dfd24c2aa5a59457a8e58fbb P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:03.185235 6483 raft_consensus.cc:374] T 66b8f583dfd24c2aa5a59457a8e58fbb P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:03.185273 6483 raft_consensus.cc:729] T 66b8f583dfd24c2aa5a59457a8e58fbb P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:46:03.185426 6483 consensus_queue.cc:260] T 66b8f583dfd24c2aa5a59457a8e58fbb P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:03.185678 6486 tablet_bootstrap.cc:492] T 66b8f583dfd24c2aa5a59457a8e58fbb P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:46:03.186311 4749 tablet_service.cc:1467] Processing CreateTablet for tablet a55d5b8b9ed04ca39888b89b82a3f833 (DEFAULT_TABLE table=rebalance_test_table_18 [id=393c8bbf6b174a9e891d96bcd09fe045]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:03.186585 4750 tablet_service.cc:1467] Processing CreateTablet for tablet c6ac59f330454d2a89ad9c11192ed4f0 (DEFAULT_TABLE table=rebalance_test_table_18 [id=393c8bbf6b174a9e891d96bcd09fe045]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:03.186823 6486 tablet_bootstrap.cc:654] T 66b8f583dfd24c2aa5a59457a8e58fbb P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:03.186604 4749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a55d5b8b9ed04ca39888b89b82a3f833. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:03.187372 4750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6ac59f330454d2a89ad9c11192ed4f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:03.188030 6482 tablet_bootstrap.cc:492] T 66b8f583dfd24c2aa5a59457a8e58fbb P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:46:03.188118 6482 ts_tablet_manager.cc:1367] T 66b8f583dfd24c2aa5a59457a8e58fbb P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 20:46:03.189391 6482 raft_consensus.cc:348] T 66b8f583dfd24c2aa5a59457a8e58fbb P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:03.189507 6482 raft_consensus.cc:374] T 66b8f583dfd24c2aa5a59457a8e58fbb P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:03.189538 6482 raft_consensus.cc:729] T 66b8f583dfd24c2aa5a59457a8e58fbb P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:46:03.189661 6482 consensus_queue.cc:260] T 66b8f583dfd24c2aa5a59457a8e58fbb P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:03.190055 6485 tablet_bootstrap.cc:492] T c6ac59f330454d2a89ad9c11192ed4f0 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:46:03.190133 6483 ts_tablet_manager.cc:1398] T 66b8f583dfd24c2aa5a59457a8e58fbb P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240502 20:46:03.190125 6482 ts_tablet_manager.cc:1398] T 66b8f583dfd24c2aa5a59457a8e58fbb P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:03.190310 6482 tablet_bootstrap.cc:492] T a55d5b8b9ed04ca39888b89b82a3f833 P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:46:03.190547 6483 tablet_bootstrap.cc:492] T a55d5b8b9ed04ca39888b89b82a3f833 P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:46:03.191267 6485 tablet_bootstrap.cc:654] T c6ac59f330454d2a89ad9c11192ed4f0 P 923cfab926e7443f8ffd7b21ad0a9d91: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:03.192483 6486 tablet_bootstrap.cc:492] T 66b8f583dfd24c2aa5a59457a8e58fbb P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:46:03.192492 6482 tablet_bootstrap.cc:654] T a55d5b8b9ed04ca39888b89b82a3f833 P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:03.192553 6483 tablet_bootstrap.cc:654] T a55d5b8b9ed04ca39888b89b82a3f833 P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:03.192690 6486 ts_tablet_manager.cc:1367] T 66b8f583dfd24c2aa5a59457a8e58fbb P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 20:46:03.193214 6486 raft_consensus.cc:348] T 66b8f583dfd24c2aa5a59457a8e58fbb P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:03.193351 6486 raft_consensus.cc:374] T 66b8f583dfd24c2aa5a59457a8e58fbb P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:03.193393 6486 raft_consensus.cc:729] T 66b8f583dfd24c2aa5a59457a8e58fbb P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:46:03.193519 6486 consensus_queue.cc:260] T 66b8f583dfd24c2aa5a59457a8e58fbb P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:03.193922 6486 ts_tablet_manager.cc:1398] T 66b8f583dfd24c2aa5a59457a8e58fbb P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:03.194207 6486 tablet_bootstrap.cc:492] T c6ac59f330454d2a89ad9c11192ed4f0 P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:46:03.194985 6483 tablet_bootstrap.cc:492] T a55d5b8b9ed04ca39888b89b82a3f833 P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:46:03.195082 6483 ts_tablet_manager.cc:1367] T a55d5b8b9ed04ca39888b89b82a3f833 P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240502 20:46:03.195132 6486 tablet_bootstrap.cc:654] T c6ac59f330454d2a89ad9c11192ed4f0 P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:03.195592 6483 raft_consensus.cc:348] T a55d5b8b9ed04ca39888b89b82a3f833 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:03.195705 6483 raft_consensus.cc:374] T a55d5b8b9ed04ca39888b89b82a3f833 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:03.195744 6483 raft_consensus.cc:729] T a55d5b8b9ed04ca39888b89b82a3f833 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:46:03.195883 6483 consensus_queue.cc:260] T a55d5b8b9ed04ca39888b89b82a3f833 P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:03.196190 6485 tablet_bootstrap.cc:492] T c6ac59f330454d2a89ad9c11192ed4f0 P 923cfab926e7443f8ffd7b21ad0a9d91: No bootstrap required, opened a new log
I20240502 20:46:03.196238 6483 ts_tablet_manager.cc:1398] T a55d5b8b9ed04ca39888b89b82a3f833 P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:03.196281 6485 ts_tablet_manager.cc:1367] T c6ac59f330454d2a89ad9c11192ed4f0 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 20:46:03.196732 6485 raft_consensus.cc:348] T c6ac59f330454d2a89ad9c11192ed4f0 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:03.196833 6482 tablet_bootstrap.cc:492] T a55d5b8b9ed04ca39888b89b82a3f833 P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:46:03.196877 6485 raft_consensus.cc:374] T c6ac59f330454d2a89ad9c11192ed4f0 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:03.196910 6485 raft_consensus.cc:729] T c6ac59f330454d2a89ad9c11192ed4f0 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: FOLLOWER
I20240502 20:46:03.196928 6482 ts_tablet_manager.cc:1367] T a55d5b8b9ed04ca39888b89b82a3f833 P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 20:46:03.197069 6485 consensus_queue.cc:260] T c6ac59f330454d2a89ad9c11192ed4f0 P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:03.197355 6485 ts_tablet_manager.cc:1398] T c6ac59f330454d2a89ad9c11192ed4f0 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:03.197398 6482 raft_consensus.cc:348] T a55d5b8b9ed04ca39888b89b82a3f833 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:03.197542 6482 raft_consensus.cc:374] T a55d5b8b9ed04ca39888b89b82a3f833 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:03.197577 6482 raft_consensus.cc:729] T a55d5b8b9ed04ca39888b89b82a3f833 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:46:03.197705 6482 consensus_queue.cc:260] T a55d5b8b9ed04ca39888b89b82a3f833 P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:03.198005 6482 ts_tablet_manager.cc:1398] T a55d5b8b9ed04ca39888b89b82a3f833 P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:03.198192 6482 tablet_bootstrap.cc:492] T c6ac59f330454d2a89ad9c11192ed4f0 P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:46:03.198211 6486 tablet_bootstrap.cc:492] T c6ac59f330454d2a89ad9c11192ed4f0 P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:46:03.198305 6486 ts_tablet_manager.cc:1367] T c6ac59f330454d2a89ad9c11192ed4f0 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 20:46:03.198804 6486 raft_consensus.cc:348] T c6ac59f330454d2a89ad9c11192ed4f0 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:03.198926 6486 raft_consensus.cc:374] T c6ac59f330454d2a89ad9c11192ed4f0 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:03.198963 6486 raft_consensus.cc:729] T c6ac59f330454d2a89ad9c11192ed4f0 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:46:03.199128 6482 tablet_bootstrap.cc:654] T c6ac59f330454d2a89ad9c11192ed4f0 P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:03.199079 6486 consensus_queue.cc:260] T c6ac59f330454d2a89ad9c11192ed4f0 P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:03.199419 6486 ts_tablet_manager.cc:1398] T c6ac59f330454d2a89ad9c11192ed4f0 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:03.199591 6486 tablet_bootstrap.cc:492] T a55d5b8b9ed04ca39888b89b82a3f833 P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:46:03.200582 6486 tablet_bootstrap.cc:654] T a55d5b8b9ed04ca39888b89b82a3f833 P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:03.202507 6482 tablet_bootstrap.cc:492] T c6ac59f330454d2a89ad9c11192ed4f0 P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:46:03.202646 6482 ts_tablet_manager.cc:1367] T c6ac59f330454d2a89ad9c11192ed4f0 P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 20:46:03.202858 6486 tablet_bootstrap.cc:492] T a55d5b8b9ed04ca39888b89b82a3f833 P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:46:03.202955 6486 ts_tablet_manager.cc:1367] T a55d5b8b9ed04ca39888b89b82a3f833 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:46:03.203184 6482 raft_consensus.cc:348] T c6ac59f330454d2a89ad9c11192ed4f0 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:03.203297 6482 raft_consensus.cc:374] T c6ac59f330454d2a89ad9c11192ed4f0 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:03.203330 6482 raft_consensus.cc:729] T c6ac59f330454d2a89ad9c11192ed4f0 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:46:03.203392 6486 raft_consensus.cc:348] T a55d5b8b9ed04ca39888b89b82a3f833 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:03.203459 6482 consensus_queue.cc:260] T c6ac59f330454d2a89ad9c11192ed4f0 P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:03.203508 6486 raft_consensus.cc:374] T a55d5b8b9ed04ca39888b89b82a3f833 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:03.203541 6486 raft_consensus.cc:729] T a55d5b8b9ed04ca39888b89b82a3f833 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:46:03.203634 6486 consensus_queue.cc:260] T a55d5b8b9ed04ca39888b89b82a3f833 P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:03.203768 6482 ts_tablet_manager.cc:1398] T c6ac59f330454d2a89ad9c11192ed4f0 P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:03.203881 6486 ts_tablet_manager.cc:1398] T a55d5b8b9ed04ca39888b89b82a3f833 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:03.223495 6475 raft_consensus.cc:482] T a55d5b8b9ed04ca39888b89b82a3f833 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:03.223636 6475 raft_consensus.cc:504] T a55d5b8b9ed04ca39888b89b82a3f833 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:03.224038 6475 leader_election.cc:288] T a55d5b8b9ed04ca39888b89b82a3f833 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097), c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:46:03.224432 4374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a55d5b8b9ed04ca39888b89b82a3f833" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2" is_pre_election: true
I20240502 20:46:03.224411 4506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a55d5b8b9ed04ca39888b89b82a3f833" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" is_pre_election: true
I20240502 20:46:03.224563 4506 raft_consensus.cc:2454] T a55d5b8b9ed04ca39888b89b82a3f833 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 0.
I20240502 20:46:03.224573 4374 raft_consensus.cc:2454] T a55d5b8b9ed04ca39888b89b82a3f833 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 0.
I20240502 20:46:03.224870 4706 leader_election.cc:302] T a55d5b8b9ed04ca39888b89b82a3f833 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:46:03.225132 6475 raft_consensus.cc:2781] T a55d5b8b9ed04ca39888b89b82a3f833 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:03.225191 6475 raft_consensus.cc:482] T a55d5b8b9ed04ca39888b89b82a3f833 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:03.225235 6475 raft_consensus.cc:3037] T a55d5b8b9ed04ca39888b89b82a3f833 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:03.226560 6475 raft_consensus.cc:504] T a55d5b8b9ed04ca39888b89b82a3f833 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:03.226900 6475 leader_election.cc:288] T a55d5b8b9ed04ca39888b89b82a3f833 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 election: Requested vote from peers c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097), c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:46:03.227200 4374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a55d5b8b9ed04ca39888b89b82a3f833" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2"
I20240502 20:46:03.227317 4374 raft_consensus.cc:3037] T a55d5b8b9ed04ca39888b89b82a3f833 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:03.227428 4506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a55d5b8b9ed04ca39888b89b82a3f833" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc"
I20240502 20:46:03.227538 4506 raft_consensus.cc:3037] T a55d5b8b9ed04ca39888b89b82a3f833 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:03.228381 4374 raft_consensus.cc:2454] T a55d5b8b9ed04ca39888b89b82a3f833 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 1.
I20240502 20:46:03.228499 4506 raft_consensus.cc:2454] T a55d5b8b9ed04ca39888b89b82a3f833 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 1.
I20240502 20:46:03.228798 4705 leader_election.cc:302] T a55d5b8b9ed04ca39888b89b82a3f833 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3d26eafd51547acb780d4aac5ebe7cc, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:46:03.229112 6475 raft_consensus.cc:2781] T a55d5b8b9ed04ca39888b89b82a3f833 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:03.229238 6475 raft_consensus.cc:686] T a55d5b8b9ed04ca39888b89b82a3f833 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Becoming Leader. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Running, Role: LEADER
I20240502 20:46:03.229408 6475 consensus_queue.cc:237] T a55d5b8b9ed04ca39888b89b82a3f833 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:03.231591 4230 catalog_manager.cc:5484] T a55d5b8b9ed04ca39888b89b82a3f833 P ed47c023452a497192f35d2f617bf9bd reported cstate change: term changed from 0 to 1, leader changed from <none> to ed47c023452a497192f35d2f617bf9bd (127.30.32.68). New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:03.267557 6238 raft_consensus.cc:482] T 66b8f583dfd24c2aa5a59457a8e58fbb P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:03.267699 6238 raft_consensus.cc:504] T 66b8f583dfd24c2aa5a59457a8e58fbb P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:03.268136 6238 leader_election.cc:288] T 66b8f583dfd24c2aa5a59457a8e58fbb P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193), c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:46:03.268548 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66b8f583dfd24c2aa5a59457a8e58fbb" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd" is_pre_election: true
I20240502 20:46:03.268630 4374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66b8f583dfd24c2aa5a59457a8e58fbb" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2" is_pre_election: true
I20240502 20:46:03.268724 4768 raft_consensus.cc:2454] T 66b8f583dfd24c2aa5a59457a8e58fbb P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 0.
I20240502 20:46:03.268761 4374 raft_consensus.cc:2454] T 66b8f583dfd24c2aa5a59457a8e58fbb P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 0.
I20240502 20:46:03.269079 4444 leader_election.cc:302] T 66b8f583dfd24c2aa5a59457a8e58fbb P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3d26eafd51547acb780d4aac5ebe7cc, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:46:03.269268 6238 raft_consensus.cc:2781] T 66b8f583dfd24c2aa5a59457a8e58fbb P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:03.269330 6238 raft_consensus.cc:482] T 66b8f583dfd24c2aa5a59457a8e58fbb P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:03.269383 6238 raft_consensus.cc:3037] T 66b8f583dfd24c2aa5a59457a8e58fbb P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:03.270193 6238 raft_consensus.cc:504] T 66b8f583dfd24c2aa5a59457a8e58fbb P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:03.270511 6238 leader_election.cc:288] T 66b8f583dfd24c2aa5a59457a8e58fbb P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 election: Requested vote from peers ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193), c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:46:03.270941 4374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66b8f583dfd24c2aa5a59457a8e58fbb" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2"
I20240502 20:46:03.270916 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66b8f583dfd24c2aa5a59457a8e58fbb" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd"
I20240502 20:46:03.271071 4768 raft_consensus.cc:3037] T 66b8f583dfd24c2aa5a59457a8e58fbb P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:03.271075 4374 raft_consensus.cc:3037] T 66b8f583dfd24c2aa5a59457a8e58fbb P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:03.271972 4374 raft_consensus.cc:2454] T 66b8f583dfd24c2aa5a59457a8e58fbb P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 1.
I20240502 20:46:03.271975 4768 raft_consensus.cc:2454] T 66b8f583dfd24c2aa5a59457a8e58fbb P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 1.
I20240502 20:46:03.272260 4444 leader_election.cc:302] T 66b8f583dfd24c2aa5a59457a8e58fbb P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3d26eafd51547acb780d4aac5ebe7cc, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:46:03.272431 6238 raft_consensus.cc:2781] T 66b8f583dfd24c2aa5a59457a8e58fbb P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:03.272518 6238 raft_consensus.cc:686] T 66b8f583dfd24c2aa5a59457a8e58fbb P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Becoming Leader. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Running, Role: LEADER
I20240502 20:46:03.272667 6238 consensus_queue.cc:237] T 66b8f583dfd24c2aa5a59457a8e58fbb P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:03.274986 4230 catalog_manager.cc:5484] T 66b8f583dfd24c2aa5a59457a8e58fbb P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: term changed from 0 to 1, leader changed from <none> to c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66). New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } }
I20240502 20:46:03.399350 6241 raft_consensus.cc:482] T c6ac59f330454d2a89ad9c11192ed4f0 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:03.399477 6241 raft_consensus.cc:504] T c6ac59f330454d2a89ad9c11192ed4f0 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:03.399883 6241 leader_election.cc:288] T c6ac59f330454d2a89ad9c11192ed4f0 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097), 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:46:03.400283 4506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac59f330454d2a89ad9c11192ed4f0" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" is_pre_election: true
I20240502 20:46:03.400283 4637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac59f330454d2a89ad9c11192ed4f0" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" is_pre_election: true
I20240502 20:46:03.400424 4637 raft_consensus.cc:2454] T c6ac59f330454d2a89ad9c11192ed4f0 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 0.
I20240502 20:46:03.400424 4506 raft_consensus.cc:2454] T c6ac59f330454d2a89ad9c11192ed4f0 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 0.
I20240502 20:46:03.400740 4703 leader_election.cc:302] T c6ac59f330454d2a89ad9c11192ed4f0 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:46:03.400916 6241 raft_consensus.cc:2781] T c6ac59f330454d2a89ad9c11192ed4f0 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:03.400975 6241 raft_consensus.cc:482] T c6ac59f330454d2a89ad9c11192ed4f0 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:03.401046 6241 raft_consensus.cc:3037] T c6ac59f330454d2a89ad9c11192ed4f0 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:03.401935 6241 raft_consensus.cc:504] T c6ac59f330454d2a89ad9c11192ed4f0 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:03.402248 6241 leader_election.cc:288] T c6ac59f330454d2a89ad9c11192ed4f0 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 election: Requested vote from peers c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097), 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:46:03.402611 4637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac59f330454d2a89ad9c11192ed4f0" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91"
I20240502 20:46:03.402653 4506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac59f330454d2a89ad9c11192ed4f0" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc"
I20240502 20:46:03.402719 4637 raft_consensus.cc:3037] T c6ac59f330454d2a89ad9c11192ed4f0 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:03.402787 4506 raft_consensus.cc:3037] T c6ac59f330454d2a89ad9c11192ed4f0 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:03.403574 4637 raft_consensus.cc:2454] T c6ac59f330454d2a89ad9c11192ed4f0 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 1.
I20240502 20:46:03.403863 4703 leader_election.cc:302] T c6ac59f330454d2a89ad9c11192ed4f0 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:46:03.403877 4506 raft_consensus.cc:2454] T c6ac59f330454d2a89ad9c11192ed4f0 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 1.
I20240502 20:46:03.404027 6241 raft_consensus.cc:2781] T c6ac59f330454d2a89ad9c11192ed4f0 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:03.404114 6241 raft_consensus.cc:686] T c6ac59f330454d2a89ad9c11192ed4f0 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Becoming Leader. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Running, Role: LEADER
I20240502 20:46:03.404273 6241 consensus_queue.cc:237] T c6ac59f330454d2a89ad9c11192ed4f0 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:03.406666 4230 catalog_manager.cc:5484] T c6ac59f330454d2a89ad9c11192ed4f0 P ed47c023452a497192f35d2f617bf9bd reported cstate change: term changed from 0 to 1, leader changed from <none> to ed47c023452a497192f35d2f617bf9bd (127.30.32.68). New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: UNKNOWN } } }
W20240502 20:46:03.546006 6498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:03.546244 6498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:03.549355 6498 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_18'
I20240502 20:46:03.634747 4374 raft_consensus.cc:1261] T a55d5b8b9ed04ca39888b89b82a3f833 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:03.634771 4638 raft_consensus.cc:1261] T c6ac59f330454d2a89ad9c11192ed4f0 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:03.635090 4506 raft_consensus.cc:1261] T c6ac59f330454d2a89ad9c11192ed4f0 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:03.635447 4508 raft_consensus.cc:1261] T a55d5b8b9ed04ca39888b89b82a3f833 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:03.635673 6529 consensus_queue.cc:1035] T c6ac59f330454d2a89ad9c11192ed4f0 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:03.635932 6475 consensus_queue.cc:1035] T a55d5b8b9ed04ca39888b89b82a3f833 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:03.636323 6475 consensus_queue.cc:1035] T a55d5b8b9ed04ca39888b89b82a3f833 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:03.636592 6241 consensus_queue.cc:1035] T c6ac59f330454d2a89ad9c11192ed4f0 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:03.659515 6414 consensus_queue.cc:1035] T 66b8f583dfd24c2aa5a59457a8e58fbb P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:03.663089 4377 raft_consensus.cc:1261] T 66b8f583dfd24c2aa5a59457a8e58fbb P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:03.663866 6414 consensus_queue.cc:1035] T 66b8f583dfd24c2aa5a59457a8e58fbb P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 118.488 ms
time per row: 0.0592438 ms
I20240502 20:46:03.711938 4230 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41812:
name: "rebalance_test_table_19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:46:03.718109 4487 tablet_service.cc:1467] Processing CreateTablet for tablet 02302fc79dc04b0396041c600c2f4bef (DEFAULT_TABLE table=rebalance_test_table_19 [id=46a52eb51c614964a91ee65db96cf854]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:03.718431 4487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02302fc79dc04b0396041c600c2f4bef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:03.718369 4618 tablet_service.cc:1467] Processing CreateTablet for tablet 45ade8c672d8445eaffb179772e3759a (DEFAULT_TABLE table=rebalance_test_table_19 [id=46a52eb51c614964a91ee65db96cf854]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:03.718647 4618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45ade8c672d8445eaffb179772e3759a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:03.718839 4619 tablet_service.cc:1467] Processing CreateTablet for tablet 02302fc79dc04b0396041c600c2f4bef (DEFAULT_TABLE table=rebalance_test_table_19 [id=46a52eb51c614964a91ee65db96cf854]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:03.719085 4619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02302fc79dc04b0396041c600c2f4bef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:03.720856 6546 tablet_bootstrap.cc:492] T 02302fc79dc04b0396041c600c2f4bef P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:46:03.721022 6547 tablet_bootstrap.cc:492] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:46:03.721352 4750 tablet_service.cc:1467] Processing CreateTablet for tablet 45ade8c672d8445eaffb179772e3759a (DEFAULT_TABLE table=rebalance_test_table_19 [id=46a52eb51c614964a91ee65db96cf854]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:03.721406 4749 tablet_service.cc:1467] Processing CreateTablet for tablet cdf218d5e45b4db19a8e2c22dbc5cc5d (DEFAULT_TABLE table=rebalance_test_table_19 [id=46a52eb51c614964a91ee65db96cf854]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:03.721657 4750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45ade8c672d8445eaffb179772e3759a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:03.721870 4749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdf218d5e45b4db19a8e2c22dbc5cc5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:03.721909 6547 tablet_bootstrap.cc:654] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:03.722050 4487 tablet_service.cc:1467] Processing CreateTablet for tablet 45ade8c672d8445eaffb179772e3759a (DEFAULT_TABLE table=rebalance_test_table_19 [id=46a52eb51c614964a91ee65db96cf854]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:03.722275 4487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45ade8c672d8445eaffb179772e3759a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:03.724468 4357 tablet_service.cc:1467] Processing CreateTablet for tablet 02302fc79dc04b0396041c600c2f4bef (DEFAULT_TABLE table=rebalance_test_table_19 [id=46a52eb51c614964a91ee65db96cf854]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:03.724603 6549 tablet_bootstrap.cc:492] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:46:03.724710 6546 tablet_bootstrap.cc:654] T 02302fc79dc04b0396041c600c2f4bef P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:03.724756 4357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02302fc79dc04b0396041c600c2f4bef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:03.725240 4619 tablet_service.cc:1467] Processing CreateTablet for tablet cdf218d5e45b4db19a8e2c22dbc5cc5d (DEFAULT_TABLE table=rebalance_test_table_19 [id=46a52eb51c614964a91ee65db96cf854]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:03.725490 4619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdf218d5e45b4db19a8e2c22dbc5cc5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:03.725523 6549 tablet_bootstrap.cc:654] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:03.725767 6546 tablet_bootstrap.cc:492] T 02302fc79dc04b0396041c600c2f4bef P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:46:03.725843 6546 ts_tablet_manager.cc:1367] T 02302fc79dc04b0396041c600c2f4bef P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 20:46:03.726445 6546 raft_consensus.cc:348] T 02302fc79dc04b0396041c600c2f4bef P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:03.726555 6546 raft_consensus.cc:374] T 02302fc79dc04b0396041c600c2f4bef P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:03.726588 6546 raft_consensus.cc:729] T 02302fc79dc04b0396041c600c2f4bef P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:46:03.726703 6546 consensus_queue.cc:260] T 02302fc79dc04b0396041c600c2f4bef P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:03.726867 6552 tablet_bootstrap.cc:492] T 02302fc79dc04b0396041c600c2f4bef P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:46:03.726979 6546 ts_tablet_manager.cc:1398] T 02302fc79dc04b0396041c600c2f4bef P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:03.727046 4355 tablet_service.cc:1467] Processing CreateTablet for tablet cdf218d5e45b4db19a8e2c22dbc5cc5d (DEFAULT_TABLE table=rebalance_test_table_19 [id=46a52eb51c614964a91ee65db96cf854]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:03.727177 6546 tablet_bootstrap.cc:492] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:46:03.727248 4355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdf218d5e45b4db19a8e2c22dbc5cc5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:03.728016 6546 tablet_bootstrap.cc:654] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:03.728443 6552 tablet_bootstrap.cc:654] T 02302fc79dc04b0396041c600c2f4bef P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:03.729387 6549 tablet_bootstrap.cc:492] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:46:03.729467 6549 ts_tablet_manager.cc:1367] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240502 20:46:03.729919 6549 raft_consensus.cc:348] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:03.730022 6549 raft_consensus.cc:374] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:03.730055 6549 raft_consensus.cc:729] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:46:03.730175 6549 consensus_queue.cc:260] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:03.729642 6547 tablet_bootstrap.cc:492] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91: No bootstrap required, opened a new log
I20240502 20:46:03.730446 6547 ts_tablet_manager.cc:1367] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 20:46:03.730544 6549 ts_tablet_manager.cc:1398] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:03.730830 6549 tablet_bootstrap.cc:492] T 45ade8c672d8445eaffb179772e3759a P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:46:03.730898 6547 raft_consensus.cc:348] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:03.730993 6547 raft_consensus.cc:374] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:03.731019 6547 raft_consensus.cc:729] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: FOLLOWER
I20240502 20:46:03.731127 6547 consensus_queue.cc:260] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:03.731726 6547 ts_tablet_manager.cc:1398] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:03.731817 6546 tablet_bootstrap.cc:492] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:46:03.731999 6547 tablet_bootstrap.cc:492] T 45ade8c672d8445eaffb179772e3759a P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:46:03.733024 6547 tablet_bootstrap.cc:654] T 45ade8c672d8445eaffb179772e3759a P 923cfab926e7443f8ffd7b21ad0a9d91: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:03.733255 6546 ts_tablet_manager.cc:1367] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 20:46:03.733726 6546 raft_consensus.cc:348] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:03.733839 6546 raft_consensus.cc:374] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:03.733866 6546 raft_consensus.cc:729] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:46:03.733991 6546 consensus_queue.cc:260] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:03.734282 6546 ts_tablet_manager.cc:1398] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:03.735309 6552 tablet_bootstrap.cc:492] T 02302fc79dc04b0396041c600c2f4bef P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:46:03.735404 6552 ts_tablet_manager.cc:1367] T 02302fc79dc04b0396041c600c2f4bef P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240502 20:46:03.735870 6552 raft_consensus.cc:348] T 02302fc79dc04b0396041c600c2f4bef P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:03.736022 6552 raft_consensus.cc:374] T 02302fc79dc04b0396041c600c2f4bef P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:03.736078 6552 raft_consensus.cc:729] T 02302fc79dc04b0396041c600c2f4bef P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:46:03.736245 6552 consensus_queue.cc:260] T 02302fc79dc04b0396041c600c2f4bef P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:03.736660 6552 ts_tablet_manager.cc:1398] T 02302fc79dc04b0396041c600c2f4bef P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:03.737038 6552 tablet_bootstrap.cc:492] T cdf218d5e45b4db19a8e2c22dbc5cc5d P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:46:03.737350 6549 tablet_bootstrap.cc:654] T 45ade8c672d8445eaffb179772e3759a P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:03.737861 6552 tablet_bootstrap.cc:654] T cdf218d5e45b4db19a8e2c22dbc5cc5d P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:03.739303 6549 tablet_bootstrap.cc:492] T 45ade8c672d8445eaffb179772e3759a P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:46:03.739398 6549 ts_tablet_manager.cc:1367] T 45ade8c672d8445eaffb179772e3759a P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 20:46:03.739411 6552 tablet_bootstrap.cc:492] T cdf218d5e45b4db19a8e2c22dbc5cc5d P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:46:03.739483 6552 ts_tablet_manager.cc:1367] T cdf218d5e45b4db19a8e2c22dbc5cc5d P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 20:46:03.739781 6549 raft_consensus.cc:348] T 45ade8c672d8445eaffb179772e3759a P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:03.739868 6549 raft_consensus.cc:374] T 45ade8c672d8445eaffb179772e3759a P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:03.739904 6549 raft_consensus.cc:729] T 45ade8c672d8445eaffb179772e3759a P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:46:03.739938 6552 raft_consensus.cc:348] T cdf218d5e45b4db19a8e2c22dbc5cc5d P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:03.740038 6552 raft_consensus.cc:374] T cdf218d5e45b4db19a8e2c22dbc5cc5d P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:03.740082 6552 raft_consensus.cc:729] T cdf218d5e45b4db19a8e2c22dbc5cc5d P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:46:03.740027 6549 consensus_queue.cc:260] T 45ade8c672d8445eaffb179772e3759a P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:03.740207 6552 consensus_queue.cc:260] T cdf218d5e45b4db19a8e2c22dbc5cc5d P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:03.740334 6549 ts_tablet_manager.cc:1398] T 45ade8c672d8445eaffb179772e3759a P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:03.740489 6552 ts_tablet_manager.cc:1398] T cdf218d5e45b4db19a8e2c22dbc5cc5d P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:03.741514 6547 tablet_bootstrap.cc:492] T 45ade8c672d8445eaffb179772e3759a P 923cfab926e7443f8ffd7b21ad0a9d91: No bootstrap required, opened a new log
I20240502 20:46:03.741604 6547 ts_tablet_manager.cc:1367] T 45ade8c672d8445eaffb179772e3759a P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240502 20:46:03.742059 6547 raft_consensus.cc:348] T 45ade8c672d8445eaffb179772e3759a P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:03.742177 6547 raft_consensus.cc:374] T 45ade8c672d8445eaffb179772e3759a P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:03.742353 6547 raft_consensus.cc:729] T 45ade8c672d8445eaffb179772e3759a P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: FOLLOWER
I20240502 20:46:03.742482 6547 consensus_queue.cc:260] T 45ade8c672d8445eaffb179772e3759a P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:03.742727 6547 ts_tablet_manager.cc:1398] T 45ade8c672d8445eaffb179772e3759a P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:03.742874 6547 tablet_bootstrap.cc:492] T cdf218d5e45b4db19a8e2c22dbc5cc5d P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:46:03.743614 6547 tablet_bootstrap.cc:654] T cdf218d5e45b4db19a8e2c22dbc5cc5d P 923cfab926e7443f8ffd7b21ad0a9d91: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:03.744539 6547 tablet_bootstrap.cc:492] T cdf218d5e45b4db19a8e2c22dbc5cc5d P 923cfab926e7443f8ffd7b21ad0a9d91: No bootstrap required, opened a new log
I20240502 20:46:03.744621 6547 ts_tablet_manager.cc:1367] T cdf218d5e45b4db19a8e2c22dbc5cc5d P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:03.745046 6547 raft_consensus.cc:348] T cdf218d5e45b4db19a8e2c22dbc5cc5d P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:03.745178 6547 raft_consensus.cc:374] T cdf218d5e45b4db19a8e2c22dbc5cc5d P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:03.745384 6547 raft_consensus.cc:729] T cdf218d5e45b4db19a8e2c22dbc5cc5d P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: FOLLOWER
I20240502 20:46:03.745813 6547 consensus_queue.cc:260] T cdf218d5e45b4db19a8e2c22dbc5cc5d P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:03.746031 6547 ts_tablet_manager.cc:1398] T cdf218d5e45b4db19a8e2c22dbc5cc5d P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:03.750383 6545 raft_consensus.cc:482] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:03.750465 6545 raft_consensus.cc:504] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:03.750758 6545 leader_election.cc:288] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097), c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:46:03.751077 4377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02302fc79dc04b0396041c600c2f4bef" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2" is_pre_election: true
I20240502 20:46:03.751211 4377 raft_consensus.cc:2454] T 02302fc79dc04b0396041c600c2f4bef P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 0.
I20240502 20:46:03.751334 4508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02302fc79dc04b0396041c600c2f4bef" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" is_pre_election: true
I20240502 20:46:03.751475 4508 raft_consensus.cc:2454] T 02302fc79dc04b0396041c600c2f4bef P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 0.
I20240502 20:46:03.751560 4575 leader_election.cc:302] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:46:03.751711 6545 raft_consensus.cc:2781] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:03.751768 6545 raft_consensus.cc:482] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:03.751811 6545 raft_consensus.cc:3037] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:03.752907 6545 raft_consensus.cc:504] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:03.753295 6545 leader_election.cc:288] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 election: Requested vote from peers c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097), c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:46:03.753585 4508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02302fc79dc04b0396041c600c2f4bef" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc"
I20240502 20:46:03.753607 4377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02302fc79dc04b0396041c600c2f4bef" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2"
I20240502 20:46:03.753706 4377 raft_consensus.cc:3037] T 02302fc79dc04b0396041c600c2f4bef P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:03.753706 4508 raft_consensus.cc:3037] T 02302fc79dc04b0396041c600c2f4bef P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:03.754627 4377 raft_consensus.cc:2454] T 02302fc79dc04b0396041c600c2f4bef P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 1.
I20240502 20:46:03.754622 4508 raft_consensus.cc:2454] T 02302fc79dc04b0396041c600c2f4bef P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 1.
I20240502 20:46:03.754887 4574 leader_election.cc:302] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, c3d26eafd51547acb780d4aac5ebe7cc; no voters:
I20240502 20:46:03.755035 6545 raft_consensus.cc:2781] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:03.755128 6545 raft_consensus.cc:686] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Becoming Leader. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Running, Role: LEADER
I20240502 20:46:03.755285 6545 consensus_queue.cc:237] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:03.757225 4229 catalog_manager.cc:5484] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67). New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:03.815943 6375 raft_consensus.cc:482] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:03.816087 6375 raft_consensus.cc:504] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:03.816540 6375 leader_election.cc:288] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141), ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:46:03.816855 4638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ade8c672d8445eaffb179772e3759a" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" is_pre_election: true
I20240502 20:46:03.816938 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ade8c672d8445eaffb179772e3759a" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd" is_pre_election: true
I20240502 20:46:03.817026 4638 raft_consensus.cc:2454] T 45ade8c672d8445eaffb179772e3759a P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 0.
I20240502 20:46:03.817118 4768 raft_consensus.cc:2454] T 45ade8c672d8445eaffb179772e3759a P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 0.
I20240502 20:46:03.817361 4441 leader_election.cc:302] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, c3d26eafd51547acb780d4aac5ebe7cc; no voters:
I20240502 20:46:03.817567 6375 raft_consensus.cc:2781] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:03.817631 6375 raft_consensus.cc:482] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:03.817658 6375 raft_consensus.cc:3037] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:03.818595 6375 raft_consensus.cc:504] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:03.818912 6375 leader_election.cc:288] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 election: Requested vote from peers 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141), ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:46:03.819216 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ade8c672d8445eaffb179772e3759a" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd"
I20240502 20:46:03.819228 4638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ade8c672d8445eaffb179772e3759a" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91"
I20240502 20:46:03.819319 4768 raft_consensus.cc:3037] T 45ade8c672d8445eaffb179772e3759a P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:03.819351 4638 raft_consensus.cc:3037] T 45ade8c672d8445eaffb179772e3759a P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:03.820127 4768 raft_consensus.cc:2454] T 45ade8c672d8445eaffb179772e3759a P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 1.
I20240502 20:46:03.820233 4638 raft_consensus.cc:2454] T 45ade8c672d8445eaffb179772e3759a P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 1.
I20240502 20:46:03.820437 4444 leader_election.cc:302] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3d26eafd51547acb780d4aac5ebe7cc, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:46:03.820628 6375 raft_consensus.cc:2781] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:03.820742 6375 raft_consensus.cc:686] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Becoming Leader. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Running, Role: LEADER
I20240502 20:46:03.820881 6375 consensus_queue.cc:237] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:03.822820 4229 catalog_manager.cc:5484] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: term changed from 0 to 1, leader changed from <none> to c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66). New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } }
I20240502 20:46:04.084929 6529 raft_consensus.cc:482] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:04.085100 6529 raft_consensus.cc:504] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:04.085573 6529 leader_election.cc:288] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141), c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:46:04.086040 4374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdf218d5e45b4db19a8e2c22dbc5cc5d" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2" is_pre_election: true
I20240502 20:46:04.086040 4638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdf218d5e45b4db19a8e2c22dbc5cc5d" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" is_pre_election: true
I20240502 20:46:04.086197 4638 raft_consensus.cc:2454] T cdf218d5e45b4db19a8e2c22dbc5cc5d P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 0.
I20240502 20:46:04.086197 4374 raft_consensus.cc:2454] T cdf218d5e45b4db19a8e2c22dbc5cc5d P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 0.
I20240502 20:46:04.086522 4703 leader_election.cc:302] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:46:04.086712 6529 raft_consensus.cc:2781] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:04.086778 6529 raft_consensus.cc:482] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:04.086805 6529 raft_consensus.cc:3037] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:04.087772 6529 raft_consensus.cc:504] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:04.088137 6529 leader_election.cc:288] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 election: Requested vote from peers 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141), c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:46:04.088598 4638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdf218d5e45b4db19a8e2c22dbc5cc5d" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91"
I20240502 20:46:04.088626 4374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdf218d5e45b4db19a8e2c22dbc5cc5d" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2"
I20240502 20:46:04.088713 4638 raft_consensus.cc:3037] T cdf218d5e45b4db19a8e2c22dbc5cc5d P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:04.088763 4374 raft_consensus.cc:3037] T cdf218d5e45b4db19a8e2c22dbc5cc5d P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:04.089726 4374 raft_consensus.cc:2454] T cdf218d5e45b4db19a8e2c22dbc5cc5d P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 1.
I20240502 20:46:04.089758 4638 raft_consensus.cc:2454] T cdf218d5e45b4db19a8e2c22dbc5cc5d P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 1.
I20240502 20:46:04.090018 4706 leader_election.cc:302] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:46:04.090210 6529 raft_consensus.cc:2781] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:04.090284 6529 raft_consensus.cc:686] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Becoming Leader. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Running, Role: LEADER
I20240502 20:46:04.090633 6529 consensus_queue.cc:237] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:04.092538 4229 catalog_manager.cc:5484] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd reported cstate change: term changed from 0 to 1, leader changed from <none> to ed47c023452a497192f35d2f617bf9bd (127.30.32.68). New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:04.197072 6545 consensus_queue.cc:1035] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:04.200682 6545 consensus_queue.cc:1035] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:04.273502 6375 consensus_queue.cc:1035] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:04.277842 6375 consensus_queue.cc:1035] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 20:46:04.303498 6562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:04.303850 6562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:04.306927 6562 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_19'
I20240502 20:46:04.403946 4638 raft_consensus.cc:1261] T cdf218d5e45b4db19a8e2c22dbc5cc5d P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:04.404067 4374 raft_consensus.cc:1261] T cdf218d5e45b4db19a8e2c22dbc5cc5d P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:04.404384 6529 consensus_queue.cc:1035] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:04.404629 6241 consensus_queue.cc:1035] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:04.417810 6605 mvcc.cc:205] Tried to move back new op lower bound from 7023340602993729536 to 7023340601716666368. Current Snapshot: MvccSnapshot[applied={T|T < 7023340602993729536}]
INSERT report
rows total: 2000
time total: 131.7 ms
time per row: 0.0658502 ms
I20240502 20:46:04.478796 30849 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskolF7xA/build/debug/bin/kudu
/tmp/dist-test-taskolF7xA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.32.69:42429
--local_ip_for_outbound_sockets=127.30.32.69
--tserver_master_addrs=127.30.32.126:40167
--webserver_port=40451
--webserver_interface=127.30.32.69
--builtin_ntp_servers=127.30.32.84:39781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 20:46:04.590970 6621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:04.591198 6621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:46:04.591250 6621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:04.594493 6621 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:46:04.594553 6621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:46:04.594646 6621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.32.69
I20240502 20:46:04.598490 6621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.32.84:39781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.32.69:42429
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.30.32.69
--webserver_port=40451
--tserver_master_addrs=127.30.32.126:40167
--never_fsync=true
--heap_profile_path=/tmp/kudu.6621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.32.69
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:46:04.599350 6621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:46:04.600708 6621 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:04.609992 6626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:04.610675 6621 server_base.cc:1034] running on GCE node
W20240502 20:46:04.611284 6629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:04.611493 6627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:04.611881 6621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:46:04.612428 6621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:46:04.613577 6621 hybrid_clock.cc:648] HybridClock initialized: now 1714682764613570 us; error 34 us; skew 500 ppm
I20240502 20:46:04.620373 6621 webserver.cc:460] Webserver started at http://127.30.32.69:40451/ using document root <none> and password file <none>
I20240502 20:46:04.621074 6621 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:04.621166 6621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:04.624853 6621 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 20:46:04.626853 6635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:04.627916 6621 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:04.628036 6621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "7845c397230d49358ba2fad9db53f97c"
format_stamp: "Formatted at 2024-05-02 20:45:49 on dist-test-slave-6336"
I20240502 20:46:04.628463 6621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:04.655728 6621 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:46:04.656415 6621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:46:04.656572 6621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:04.657155 6621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:04.658030 6621 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:04.658085 6621 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:04.658124 6621 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:04.658150 6621 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:04.666842 6621 rpc_server.cc:307] RPC server started. Bound to: 127.30.32.69:42429
I20240502 20:46:04.666867 6748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.32.69:42429 every 8 connection(s)
I20240502 20:46:04.667841 6621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 20:46:04.674722 30849 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskolF7xA/build/debug/bin/kudu as pid 6621
I20240502 20:46:04.675184 30849 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskolF7xA/build/debug/bin/kudu
/tmp/dist-test-taskolF7xA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.32.70:41867
--local_ip_for_outbound_sockets=127.30.32.70
--tserver_master_addrs=127.30.32.126:40167
--webserver_port=42737
--webserver_interface=127.30.32.70
--builtin_ntp_servers=127.30.32.84:39781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 20:46:04.677387 6749 heartbeater.cc:347] Connected to a master server at 127.30.32.126:40167
I20240502 20:46:04.677626 6749 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:04.678249 6749 heartbeater.cc:511] Master 127.30.32.126:40167 requested a full tablet report, sending...
I20240502 20:46:04.679303 4233 ts_manager.cc:194] Re-registered known tserver with Master: 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
I20240502 20:46:04.679904 4233 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.32.69:54121
W20240502 20:46:04.790859 6752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:04.791085 6752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:46:04.791162 6752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:04.794206 6752 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:46:04.794279 6752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:46:04.794373 6752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.32.70
I20240502 20:46:04.797971 6752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.32.84:39781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.32.70:41867
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.30.32.70
--webserver_port=42737
--tserver_master_addrs=127.30.32.126:40167
--never_fsync=true
--heap_profile_path=/tmp/kudu.6752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.32.70
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:46:04.798882 6752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:46:04.799998 6752 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:04.809078 6761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:04.809182 6759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:04.809870 6758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:04.810572 6752 server_base.cc:1034] running on GCE node
I20240502 20:46:04.810979 6752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:46:04.811506 6752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:46:04.812647 6752 hybrid_clock.cc:648] HybridClock initialized: now 1714682764812633 us; error 26 us; skew 500 ppm
I20240502 20:46:04.814817 6752 webserver.cc:460] Webserver started at http://127.30.32.70:42737/ using document root <none> and password file <none>
I20240502 20:46:04.815472 6752 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:04.815564 6752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:04.819247 6752 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20240502 20:46:04.821158 6767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:04.822114 6752 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 20:46:04.822204 6752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "a4bf1df7fa6b4948be502c182bfbb833"
format_stamp: "Formatted at 2024-05-02 20:45:49 on dist-test-slave-6336"
I20240502 20:46:04.822569 6752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:04.838150 6752 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:46:04.838723 6752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:46:04.838856 6752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:04.839325 6752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:04.840154 6752 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:04.840199 6752 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:04.840237 6752 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:04.840260 6752 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:04.850677 6752 rpc_server.cc:307] RPC server started. Bound to: 127.30.32.70:41867
I20240502 20:46:04.850703 6880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.32.70:41867 every 8 connection(s)
I20240502 20:46:04.851902 6752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 20:46:04.861840 30849 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskolF7xA/build/debug/bin/kudu as pid 6752
I20240502 20:46:04.862321 30849 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskolF7xA/build/debug/bin/kudu
/tmp/dist-test-taskolF7xA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.32.71:46841
--local_ip_for_outbound_sockets=127.30.32.71
--tserver_master_addrs=127.30.32.126:40167
--webserver_port=35103
--webserver_interface=127.30.32.71
--builtin_ntp_servers=127.30.32.84:39781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 20:46:04.862502 6881 heartbeater.cc:347] Connected to a master server at 127.30.32.126:40167
I20240502 20:46:04.862723 6881 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:04.863319 6881 heartbeater.cc:511] Master 127.30.32.126:40167 requested a full tablet report, sending...
I20240502 20:46:04.864435 4233 ts_manager.cc:194] Re-registered known tserver with Master: a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867)
I20240502 20:46:04.865167 4233 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.32.70:55267
W20240502 20:46:04.980245 6885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:04.980469 6885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:46:04.980547 6885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:04.983714 6885 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:46:04.983785 6885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:46:04.983884 6885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.32.71
I20240502 20:46:04.987896 6885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.32.84:39781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.32.71:46841
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.30.32.71
--webserver_port=35103
--tserver_master_addrs=127.30.32.126:40167
--never_fsync=true
--heap_profile_path=/tmp/kudu.6885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.32.71
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:46:04.988840 6885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:46:04.990115 6885 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:04.998648 6891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:04.999567 6890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:05.000357 6885 server_base.cc:1034] running on GCE node
W20240502 20:46:05.000516 6893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:05.000888 6885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:46:05.001535 6885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:46:05.002701 6885 hybrid_clock.cc:648] HybridClock initialized: now 1714682765002684 us; error 26 us; skew 500 ppm
I20240502 20:46:05.004753 6885 webserver.cc:460] Webserver started at http://127.30.32.71:35103/ using document root <none> and password file <none>
I20240502 20:46:05.005369 6885 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:05.005455 6885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:05.009243 6885 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:05.011160 6899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:05.012144 6885 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:05.012274 6885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "a47e5c56f13843018ac584914a8d4707"
format_stamp: "Formatted at 2024-05-02 20:45:49 on dist-test-slave-6336"
I20240502 20:46:05.012709 6885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:05.037571 6885 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:46:05.038266 6885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:46:05.038435 6885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:05.039032 6885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:05.040109 6885 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:05.040170 6885 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:05.040221 6885 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:05.040258 6885 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:05.049818 6885 rpc_server.cc:307] RPC server started. Bound to: 127.30.32.71:46841
I20240502 20:46:05.049847 7012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.32.71:46841 every 8 connection(s)
I20240502 20:46:05.050918 6885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 20:46:05.059491 7013 heartbeater.cc:347] Connected to a master server at 127.30.32.126:40167
I20240502 20:46:05.059722 7013 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:05.060179 30849 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskolF7xA/build/debug/bin/kudu as pid 6885
I20240502 20:46:05.060303 7013 heartbeater.cc:511] Master 127.30.32.126:40167 requested a full tablet report, sending...
I20240502 20:46:05.060600 30849 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskolF7xA/build/debug/bin/kudu
/tmp/dist-test-taskolF7xA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.32.72:44643
--local_ip_for_outbound_sockets=127.30.32.72
--tserver_master_addrs=127.30.32.126:40167
--webserver_port=38697
--webserver_interface=127.30.32.72
--builtin_ntp_servers=127.30.32.84:39781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 20:46:05.061403 4233 ts_manager.cc:194] Re-registered known tserver with Master: a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
I20240502 20:46:05.062012 4233 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.32.71:48719
W20240502 20:46:05.173882 7016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:05.174091 7016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:46:05.174160 7016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:05.177343 7016 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:46:05.177420 7016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:46:05.177531 7016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.32.72
I20240502 20:46:05.181855 7016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.32.84:39781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.32.72:44643
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.30.32.72
--webserver_port=38697
--tserver_master_addrs=127.30.32.126:40167
--never_fsync=true
--heap_profile_path=/tmp/kudu.7016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.32.72
--log_dir=/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:46:05.182993 7016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:46:05.184077 7016 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:05.192155 7022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:05.193004 7025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:05.193771 7023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:05.193841 7016 server_base.cc:1034] running on GCE node
I20240502 20:46:05.194183 7016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:46:05.194660 7016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:46:05.195822 7016 hybrid_clock.cc:648] HybridClock initialized: now 1714682765195811 us; error 25 us; skew 500 ppm
I20240502 20:46:05.197643 7016 webserver.cc:460] Webserver started at http://127.30.32.72:38697/ using document root <none> and password file <none>
I20240502 20:46:05.198251 7016 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:05.198339 7016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:05.202025 7016 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20240502 20:46:05.204142 7031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:05.205207 7016 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:05.205307 7016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "201d2b6485084e7dad2cbb722af94779"
format_stamp: "Formatted at 2024-05-02 20:45:50 on dist-test-slave-6336"
I20240502 20:46:05.205775 7016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:05.217209 7016 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:46:05.217784 7016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:46:05.217929 7016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:05.218469 7016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:05.219437 7016 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:05.219493 7016 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:05.219537 7016 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:05.219565 7016 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:05.229220 7016 rpc_server.cc:307] RPC server started. Bound to: 127.30.32.72:44643
I20240502 20:46:05.229251 7144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.32.72:44643 every 8 connection(s)
I20240502 20:46:05.230386 7016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 20:46:05.236867 30849 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskolF7xA/build/debug/bin/kudu as pid 7016
I20240502 20:46:05.240479 4233 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41812:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 20:46:05.244107 7145 heartbeater.cc:347] Connected to a master server at 127.30.32.126:40167
I20240502 20:46:05.244385 7145 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:05.245112 7145 heartbeater.cc:511] Master 127.30.32.126:40167 requested a full tablet report, sending...
I20240502 20:46:05.246299 4234 ts_manager.cc:194] Re-registered known tserver with Master: 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
I20240502 20:46:05.247140 4234 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.32.72:45873
I20240502 20:46:05.253773 4487 tablet_service.cc:1467] Processing CreateTablet for tablet 125186c6a01347dd96c1b111cbe8b64e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=67b3a17c6cdb49808f2f3fde61ef1f54]), partition=
I20240502 20:46:05.254056 4487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 125186c6a01347dd96c1b111cbe8b64e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:05.260586 6683 tablet_service.cc:1467] Processing CreateTablet for tablet 125186c6a01347dd96c1b111cbe8b64e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=67b3a17c6cdb49808f2f3fde61ef1f54]), partition=
I20240502 20:46:05.261369 6683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 125186c6a01347dd96c1b111cbe8b64e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:05.268167 7162 tablet_bootstrap.cc:492] T 125186c6a01347dd96c1b111cbe8b64e P 7845c397230d49358ba2fad9db53f97c: Bootstrap starting.
I20240502 20:46:05.269203 7161 tablet_bootstrap.cc:492] T 125186c6a01347dd96c1b111cbe8b64e P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:46:05.270145 7161 tablet_bootstrap.cc:654] T 125186c6a01347dd96c1b111cbe8b64e P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:05.270690 7162 tablet_bootstrap.cc:654] T 125186c6a01347dd96c1b111cbe8b64e P 7845c397230d49358ba2fad9db53f97c: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:05.271337 7161 tablet_bootstrap.cc:492] T 125186c6a01347dd96c1b111cbe8b64e P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:46:05.271517 7162 log.cc:824] T 125186c6a01347dd96c1b111cbe8b64e P 7845c397230d49358ba2fad9db53f97c: Log is configured to *not* fsync() on all Append() calls
I20240502 20:46:05.272116 7161 ts_tablet_manager.cc:1367] T 125186c6a01347dd96c1b111cbe8b64e P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:46:05.272665 7161 raft_consensus.cc:348] T 125186c6a01347dd96c1b111cbe8b64e P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:05.272914 7161 raft_consensus.cc:374] T 125186c6a01347dd96c1b111cbe8b64e P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:05.273041 7161 raft_consensus.cc:729] T 125186c6a01347dd96c1b111cbe8b64e P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:46:05.273260 7161 consensus_queue.cc:260] T 125186c6a01347dd96c1b111cbe8b64e P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:05.274137 7161 ts_tablet_manager.cc:1398] T 125186c6a01347dd96c1b111cbe8b64e P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:05.275074 7162 tablet_bootstrap.cc:492] T 125186c6a01347dd96c1b111cbe8b64e P 7845c397230d49358ba2fad9db53f97c: No bootstrap required, opened a new log
I20240502 20:46:05.275261 7162 ts_tablet_manager.cc:1367] T 125186c6a01347dd96c1b111cbe8b64e P 7845c397230d49358ba2fad9db53f97c: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20240502 20:46:05.276265 6809 tablet_service.cc:1467] Processing CreateTablet for tablet 125186c6a01347dd96c1b111cbe8b64e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=67b3a17c6cdb49808f2f3fde61ef1f54]), partition=
I20240502 20:46:05.276965 6809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 125186c6a01347dd96c1b111cbe8b64e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:05.278949 7162 raft_consensus.cc:348] T 125186c6a01347dd96c1b111cbe8b64e P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:05.279162 7162 raft_consensus.cc:374] T 125186c6a01347dd96c1b111cbe8b64e P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:05.279224 7162 raft_consensus.cc:729] T 125186c6a01347dd96c1b111cbe8b64e P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Initialized, Role: FOLLOWER
I20240502 20:46:05.280004 7162 consensus_queue.cc:260] T 125186c6a01347dd96c1b111cbe8b64e P 7845c397230d49358ba2fad9db53f97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:05.280864 7162 ts_tablet_manager.cc:1398] T 125186c6a01347dd96c1b111cbe8b64e P 7845c397230d49358ba2fad9db53f97c: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20240502 20:46:05.281412 6749 heartbeater.cc:503] Master 127.30.32.126:40167 was elected leader, sending a full tablet report...
I20240502 20:46:05.285250 7166 tablet_bootstrap.cc:492] T 125186c6a01347dd96c1b111cbe8b64e P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap starting.
I20240502 20:46:05.287402 7166 tablet_bootstrap.cc:654] T 125186c6a01347dd96c1b111cbe8b64e P a4bf1df7fa6b4948be502c182bfbb833: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:05.288049 7166 log.cc:824] T 125186c6a01347dd96c1b111cbe8b64e P a4bf1df7fa6b4948be502c182bfbb833: Log is configured to *not* fsync() on all Append() calls
I20240502 20:46:05.289724 7166 tablet_bootstrap.cc:492] T 125186c6a01347dd96c1b111cbe8b64e P a4bf1df7fa6b4948be502c182bfbb833: No bootstrap required, opened a new log
I20240502 20:46:05.289908 7166 ts_tablet_manager.cc:1367] T 125186c6a01347dd96c1b111cbe8b64e P a4bf1df7fa6b4948be502c182bfbb833: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 20:46:05.293494 7166 raft_consensus.cc:348] T 125186c6a01347dd96c1b111cbe8b64e P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:05.293690 7166 raft_consensus.cc:374] T 125186c6a01347dd96c1b111cbe8b64e P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:05.293757 7166 raft_consensus.cc:729] T 125186c6a01347dd96c1b111cbe8b64e P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Initialized, Role: FOLLOWER
I20240502 20:46:05.294165 7166 consensus_queue.cc:260] T 125186c6a01347dd96c1b111cbe8b64e P a4bf1df7fa6b4948be502c182bfbb833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:05.294745 6881 heartbeater.cc:503] Master 127.30.32.126:40167 was elected leader, sending a full tablet report...
I20240502 20:46:05.294961 7166 ts_tablet_manager.cc:1398] T 125186c6a01347dd96c1b111cbe8b64e P a4bf1df7fa6b4948be502c182bfbb833: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20240502 20:46:05.353205 6882 tablet.cc:2369] T 125186c6a01347dd96c1b111cbe8b64e P a4bf1df7fa6b4948be502c182bfbb833: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 20:46:05.400557 7168 raft_consensus.cc:482] T 125186c6a01347dd96c1b111cbe8b64e P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:05.400729 7168 raft_consensus.cc:504] T 125186c6a01347dd96c1b111cbe8b64e P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:05.401858 7168 leader_election.cc:288] T 125186c6a01347dd96c1b111cbe8b64e P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097)
I20240502 20:46:05.406369 6703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125186c6a01347dd96c1b111cbe8b64e" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7845c397230d49358ba2fad9db53f97c" is_pre_election: true
I20240502 20:46:05.406651 6703 raft_consensus.cc:2454] T 125186c6a01347dd96c1b111cbe8b64e P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 0.
I20240502 20:46:05.407125 6770 leader_election.cc:302] T 125186c6a01347dd96c1b111cbe8b64e P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7845c397230d49358ba2fad9db53f97c, a4bf1df7fa6b4948be502c182bfbb833; no voters:
I20240502 20:46:05.407367 7168 raft_consensus.cc:2781] T 125186c6a01347dd96c1b111cbe8b64e P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:05.407442 7168 raft_consensus.cc:482] T 125186c6a01347dd96c1b111cbe8b64e P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:05.407491 7168 raft_consensus.cc:3037] T 125186c6a01347dd96c1b111cbe8b64e P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:05.408568 7168 raft_consensus.cc:504] T 125186c6a01347dd96c1b111cbe8b64e P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:05.408948 7168 leader_election.cc:288] T 125186c6a01347dd96c1b111cbe8b64e P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 election: Requested vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097)
I20240502 20:46:05.409092 4506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125186c6a01347dd96c1b111cbe8b64e" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" is_pre_election: true
I20240502 20:46:05.409222 4506 raft_consensus.cc:2454] T 125186c6a01347dd96c1b111cbe8b64e P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 0.
I20240502 20:46:05.409309 6703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125186c6a01347dd96c1b111cbe8b64e" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7845c397230d49358ba2fad9db53f97c"
I20240502 20:46:05.409485 6703 raft_consensus.cc:3037] T 125186c6a01347dd96c1b111cbe8b64e P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:05.409559 4506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125186c6a01347dd96c1b111cbe8b64e" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc"
I20240502 20:46:05.409657 4506 raft_consensus.cc:3037] T 125186c6a01347dd96c1b111cbe8b64e P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:05.410522 6703 raft_consensus.cc:2454] T 125186c6a01347dd96c1b111cbe8b64e P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 1.
I20240502 20:46:05.410692 4506 raft_consensus.cc:2454] T 125186c6a01347dd96c1b111cbe8b64e P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 1.
I20240502 20:46:05.410846 6770 leader_election.cc:302] T 125186c6a01347dd96c1b111cbe8b64e P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7845c397230d49358ba2fad9db53f97c, a4bf1df7fa6b4948be502c182bfbb833; no voters:
I20240502 20:46:05.411023 7168 raft_consensus.cc:2781] T 125186c6a01347dd96c1b111cbe8b64e P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:05.411216 7168 raft_consensus.cc:686] T 125186c6a01347dd96c1b111cbe8b64e P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEADER]: Becoming Leader. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Running, Role: LEADER
I20240502 20:46:05.411542 7168 consensus_queue.cc:237] T 125186c6a01347dd96c1b111cbe8b64e P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:05.414336 4234 catalog_manager.cc:5484] T 125186c6a01347dd96c1b111cbe8b64e P a4bf1df7fa6b4948be502c182bfbb833 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70). New cstate: current_term: 1 leader_uuid: "a4bf1df7fa6b4948be502c182bfbb833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: UNKNOWN } } }
W20240502 20:46:05.419026 6750 tablet.cc:2369] T 125186c6a01347dd96c1b111cbe8b64e P 7845c397230d49358ba2fad9db53f97c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 20:46:05.499506 6809 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:05.503036 7079 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:05.511817 6683 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:05.514356 4619 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 20:46:05.515446 4750 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 20:46:05.516809 6947 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:05.526775 4355 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 20:46:05.527158 4487 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 20:46:05.550235 4234 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 20:46:05.563203 4506 raft_consensus.cc:1261] T 125186c6a01347dd96c1b111cbe8b64e P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer a4bf1df7fa6b4948be502c182bfbb833: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:05.563678 6703 raft_consensus.cc:1261] T 125186c6a01347dd96c1b111cbe8b64e P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Refusing update from remote peer a4bf1df7fa6b4948be502c182bfbb833: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:05.563800 7186 consensus_queue.cc:1035] T 125186c6a01347dd96c1b111cbe8b64e P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:05.564635 7186 consensus_queue.cc:1035] T 125186c6a01347dd96c1b111cbe8b64e P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:05.567247 4234 catalog_manager.cc:5576] TS c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097) does not have the latest schema for tablet 125186c6a01347dd96c1b111cbe8b64e (table rebalancer_dynamic_table [id=67b3a17c6cdb49808f2f3fde61ef1f54]). Expected version 1 got 0
I20240502 20:46:05.568444 7234 tablet.cc:1722] T 125186c6a01347dd96c1b111cbe8b64e P a4bf1df7fa6b4948be502c182bfbb833: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 20:46:05.568922 4234 catalog_manager.cc:5576] TS a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867) does not have the latest schema for tablet 125186c6a01347dd96c1b111cbe8b64e (table rebalancer_dynamic_table [id=67b3a17c6cdb49808f2f3fde61ef1f54]). Expected version 1 got 0
I20240502 20:46:05.569756 4229 catalog_manager.cc:5576] TS c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097) does not have the latest schema for tablet 125186c6a01347dd96c1b111cbe8b64e (table rebalancer_dynamic_table [id=67b3a17c6cdb49808f2f3fde61ef1f54]). Expected version 1 got 0
I20240502 20:46:05.570676 7230 mvcc.cc:205] Tried to move back new op lower bound from 7023340607740084224 to 7023340607128313856. Current Snapshot: MvccSnapshot[applied={T|T < 7023340607740084224}]
I20240502 20:46:05.570727 4229 catalog_manager.cc:5576] TS 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429) does not have the latest schema for tablet 125186c6a01347dd96c1b111cbe8b64e (table rebalancer_dynamic_table [id=67b3a17c6cdb49808f2f3fde61ef1f54]). Expected version 1 got 0
I20240502 20:46:05.571345 7233 tablet.cc:1722] T 125186c6a01347dd96c1b111cbe8b64e P c3d26eafd51547acb780d4aac5ebe7cc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 20:46:05.571810 7234 tablet.cc:1620] T 125186c6a01347dd96c1b111cbe8b64e P a4bf1df7fa6b4948be502c182bfbb833: MemRowSet was empty: no flush needed.
I20240502 20:46:05.572072 7233 tablet.cc:1620] T 125186c6a01347dd96c1b111cbe8b64e P c3d26eafd51547acb780d4aac5ebe7cc: MemRowSet was empty: no flush needed.
I20240502 20:46:05.578781 4229 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: rebalancer_dynamic_table [id=67b3a17c6cdb49808f2f3fde61ef1f54] alter complete (version 1)
I20240502 20:46:05.586213 7235 mvcc.cc:205] Tried to move back new op lower bound from 7023340607740084224 to 7023340607128313856. Current Snapshot: MvccSnapshot[applied={T|T < 7023340607740084224}]
I20240502 20:46:05.587819 4229 catalog_manager.cc:5576] TS 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429) does not have the latest schema for tablet 125186c6a01347dd96c1b111cbe8b64e (table rebalancer_dynamic_table [id=67b3a17c6cdb49808f2f3fde61ef1f54]). Expected version 1 got 0
I20240502 20:46:05.589155 7242 tablet.cc:1722] T 125186c6a01347dd96c1b111cbe8b64e P 7845c397230d49358ba2fad9db53f97c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 20:46:05.589936 7242 tablet.cc:1620] T 125186c6a01347dd96c1b111cbe8b64e P 7845c397230d49358ba2fad9db53f97c: MemRowSet was empty: no flush needed.
I20240502 20:46:05.684758 4234 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 20:46:05.691201 7234 tablet.cc:1722] T 125186c6a01347dd96c1b111cbe8b64e P a4bf1df7fa6b4948be502c182bfbb833: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 20:46:05.691538 7234 tablet.cc:1620] T 125186c6a01347dd96c1b111cbe8b64e P a4bf1df7fa6b4948be502c182bfbb833: MemRowSet was empty: no flush needed.
I20240502 20:46:05.693110 7233 tablet.cc:1722] T 125186c6a01347dd96c1b111cbe8b64e P c3d26eafd51547acb780d4aac5ebe7cc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 20:46:05.693408 7233 tablet.cc:1620] T 125186c6a01347dd96c1b111cbe8b64e P c3d26eafd51547acb780d4aac5ebe7cc: MemRowSet was empty: no flush needed.
I20240502 20:46:05.695313 7242 tablet.cc:1722] T 125186c6a01347dd96c1b111cbe8b64e P 7845c397230d49358ba2fad9db53f97c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 20:46:05.695611 7242 tablet.cc:1620] T 125186c6a01347dd96c1b111cbe8b64e P 7845c397230d49358ba2fad9db53f97c: MemRowSet was empty: no flush needed.
I20240502 20:46:05.698541 4227 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: rebalancer_dynamic_table [id=67b3a17c6cdb49808f2f3fde61ef1f54] alter complete (version 2)
I20240502 20:46:05.707258 6683 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:05.709821 7079 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:05.717767 6947 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:05.718245 6809 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:05.722774 4355 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 20:46:05.732621 4487 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 20:46:05.734700 4750 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 20:46:05.738214 4230 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41812:
name: "rebalancer_extra_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:46:05.740198 4234 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240502 20:46:05.740325 4234 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240502 20:46:05.746551 4234 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: Sending DeleteTablet for 3 replicas of tablet 3cb28ecff16240afb413a1d0e51733b0
I20240502 20:46:05.746905 4234 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: Sending DeleteTablet for 3 replicas of tablet 48c3fb2ce0fc41a4a12a757052beccb8
I20240502 20:46:05.747089 4234 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: Sending DeleteTablet for 3 replicas of tablet 84a90777f631447d8ba3ffb720d370e2
I20240502 20:46:05.747961 4487 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cb28ecff16240afb413a1d0e51733b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:05 UTC) from {username='slave'} at 127.0.0.1:48890
I20240502 20:46:05.748064 4619 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cb28ecff16240afb413a1d0e51733b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:05 UTC) from {username='slave'} at 127.0.0.1:57068
I20240502 20:46:05.748454 4488 tablet_service.cc:1514] Processing DeleteTablet for tablet 48c3fb2ce0fc41a4a12a757052beccb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:05 UTC) from {username='slave'} at 127.0.0.1:48890
I20240502 20:46:05.749416 4357 tablet_service.cc:1514] Processing DeleteTablet for tablet 48c3fb2ce0fc41a4a12a757052beccb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:05 UTC) from {username='slave'} at 127.0.0.1:38306
I20240502 20:46:05.749505 4355 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cb28ecff16240afb413a1d0e51733b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:05 UTC) from {username='slave'} at 127.0.0.1:38306
I20240502 20:46:05.749676 4618 tablet_service.cc:1514] Processing DeleteTablet for tablet 84a90777f631447d8ba3ffb720d370e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:05 UTC) from {username='slave'} at 127.0.0.1:57068
I20240502 20:46:05.749743 4356 tablet_service.cc:1514] Processing DeleteTablet for tablet 84a90777f631447d8ba3ffb720d370e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:05 UTC) from {username='slave'} at 127.0.0.1:38306
I20240502 20:46:05.750424 4354 tablet_service.cc:1467] Processing CreateTablet for tablet 8bbf33f0ec4a4adb9a979b08606ee822 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=09a2b90df269425ba3ab97f05a1fc119]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:05.750699 4354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bbf33f0ec4a4adb9a979b08606ee822. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:05.750633 6809 tablet_service.cc:1467] Processing CreateTablet for tablet d6c2626b3ee7493c8a349f08e17082b5 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=09a2b90df269425ba3ab97f05a1fc119]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:05.750701 6683 tablet_service.cc:1467] Processing CreateTablet for tablet bcfd626ea0a04fa7b03dd0ede6b35266 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=09a2b90df269425ba3ab97f05a1fc119]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:05.750977 6809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6c2626b3ee7493c8a349f08e17082b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:05.751044 6683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcfd626ea0a04fa7b03dd0ede6b35266. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:05.753332 4750 tablet_service.cc:1514] Processing DeleteTablet for tablet 48c3fb2ce0fc41a4a12a757052beccb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:05 UTC) from {username='slave'} at 127.0.0.1:38526
I20240502 20:46:05.753551 4749 tablet_service.cc:1514] Processing DeleteTablet for tablet 84a90777f631447d8ba3ffb720d370e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:05 UTC) from {username='slave'} at 127.0.0.1:38526
I20240502 20:46:05.755380 7162 tablet_bootstrap.cc:492] T bcfd626ea0a04fa7b03dd0ede6b35266 P 7845c397230d49358ba2fad9db53f97c: Bootstrap starting.
I20240502 20:46:05.755398 7297 tablet_replica.cc:330] T 3cb28ecff16240afb413a1d0e51733b0 P c92e6e46782849479c8a0c6d82276fb2: stopping tablet replica
I20240502 20:46:05.755686 7297 raft_consensus.cc:2229] T 3cb28ecff16240afb413a1d0e51733b0 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:05.755872 7297 raft_consensus.cc:2258] T 3cb28ecff16240afb413a1d0e51733b0 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:05.756191 7162 tablet_bootstrap.cc:654] T bcfd626ea0a04fa7b03dd0ede6b35266 P 7845c397230d49358ba2fad9db53f97c: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:05.757043 7297 ts_tablet_manager.cc:1875] T 3cb28ecff16240afb413a1d0e51733b0 P c92e6e46782849479c8a0c6d82276fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:05.759178 7166 tablet_bootstrap.cc:492] T d6c2626b3ee7493c8a349f08e17082b5 P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap starting.
I20240502 20:46:05.759440 7297 ts_tablet_manager.cc:1888] T 3cb28ecff16240afb413a1d0e51733b0 P c92e6e46782849479c8a0c6d82276fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 20:46:05.759524 7297 log.cc:1196] T 3cb28ecff16240afb413a1d0e51733b0 P c92e6e46782849479c8a0c6d82276fb2: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal/wals/3cb28ecff16240afb413a1d0e51733b0
I20240502 20:46:05.759948 7297 ts_tablet_manager.cc:1909] T 3cb28ecff16240afb413a1d0e51733b0 P c92e6e46782849479c8a0c6d82276fb2: Deleting consensus metadata
I20240502 20:46:05.760460 7166 tablet_bootstrap.cc:654] T d6c2626b3ee7493c8a349f08e17082b5 P a4bf1df7fa6b4948be502c182bfbb833: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:05.761164 4222 catalog_manager.cc:4830] TS c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827): tablet 3cb28ecff16240afb413a1d0e51733b0 (table rebalance_test_table_0 [id=9a093bb257b04f07a867d916725eb265]) successfully deleted
I20240502 20:46:05.761435 7297 tablet_replica.cc:330] T 84a90777f631447d8ba3ffb720d370e2 P c92e6e46782849479c8a0c6d82276fb2: stopping tablet replica
I20240502 20:46:05.761512 4618 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 20:46:05.761559 7297 raft_consensus.cc:2229] T 84a90777f631447d8ba3ffb720d370e2 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:05.761674 7297 raft_consensus.cc:2258] T 84a90777f631447d8ba3ffb720d370e2 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:05.762137 7297 ts_tablet_manager.cc:1875] T 84a90777f631447d8ba3ffb720d370e2 P c92e6e46782849479c8a0c6d82276fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:05.763492 4618 tablet_service.cc:1467] Processing CreateTablet for tablet d6c2626b3ee7493c8a349f08e17082b5 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=09a2b90df269425ba3ab97f05a1fc119]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:05.763767 4618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6c2626b3ee7493c8a349f08e17082b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:05.764163 7297 ts_tablet_manager.cc:1888] T 84a90777f631447d8ba3ffb720d370e2 P c92e6e46782849479c8a0c6d82276fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 20:46:05.764238 7297 log.cc:1196] T 84a90777f631447d8ba3ffb720d370e2 P c92e6e46782849479c8a0c6d82276fb2: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal/wals/84a90777f631447d8ba3ffb720d370e2
I20240502 20:46:05.764580 7297 ts_tablet_manager.cc:1909] T 84a90777f631447d8ba3ffb720d370e2 P c92e6e46782849479c8a0c6d82276fb2: Deleting consensus metadata
I20240502 20:46:05.765237 4222 catalog_manager.cc:4830] TS c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827): tablet 84a90777f631447d8ba3ffb720d370e2 (table rebalance_test_table_0 [id=9a093bb257b04f07a867d916725eb265]) successfully deleted
I20240502 20:46:05.765491 7297 tablet_replica.cc:330] T 48c3fb2ce0fc41a4a12a757052beccb8 P c92e6e46782849479c8a0c6d82276fb2: stopping tablet replica
I20240502 20:46:05.765619 7297 raft_consensus.cc:2229] T 48c3fb2ce0fc41a4a12a757052beccb8 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:05.765729 7297 raft_consensus.cc:2258] T 48c3fb2ce0fc41a4a12a757052beccb8 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:05.766165 7297 ts_tablet_manager.cc:1875] T 48c3fb2ce0fc41a4a12a757052beccb8 P c92e6e46782849479c8a0c6d82276fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:05.766932 6946 tablet_service.cc:1467] Processing CreateTablet for tablet d6c2626b3ee7493c8a349f08e17082b5 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=09a2b90df269425ba3ab97f05a1fc119]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:05.767931 6946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6c2626b3ee7493c8a349f08e17082b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:05.768345 7297 ts_tablet_manager.cc:1888] T 48c3fb2ce0fc41a4a12a757052beccb8 P c92e6e46782849479c8a0c6d82276fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 20:46:05.768458 7297 log.cc:1196] T 48c3fb2ce0fc41a4a12a757052beccb8 P c92e6e46782849479c8a0c6d82276fb2: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal/wals/48c3fb2ce0fc41a4a12a757052beccb8
I20240502 20:46:05.768757 7297 ts_tablet_manager.cc:1909] T 48c3fb2ce0fc41a4a12a757052beccb8 P c92e6e46782849479c8a0c6d82276fb2: Deleting consensus metadata
I20240502 20:46:05.769148 6947 tablet_service.cc:1467] Processing CreateTablet for tablet 8bbf33f0ec4a4adb9a979b08606ee822 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=09a2b90df269425ba3ab97f05a1fc119]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:05.769480 4222 catalog_manager.cc:4830] TS c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827): tablet 48c3fb2ce0fc41a4a12a757052beccb8 (table rebalance_test_table_0 [id=9a093bb257b04f07a867d916725eb265]) successfully deleted
I20240502 20:46:05.769479 6945 tablet_service.cc:1467] Processing CreateTablet for tablet bcfd626ea0a04fa7b03dd0ede6b35266 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=09a2b90df269425ba3ab97f05a1fc119]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:05.770984 6947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bbf33f0ec4a4adb9a979b08606ee822. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:05.773351 7078 tablet_service.cc:1467] Processing CreateTablet for tablet bcfd626ea0a04fa7b03dd0ede6b35266 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=09a2b90df269425ba3ab97f05a1fc119]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:05.774051 7078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcfd626ea0a04fa7b03dd0ede6b35266. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:05.776737 6945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcfd626ea0a04fa7b03dd0ede6b35266. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:05.772711 7079 tablet_service.cc:1467] Processing CreateTablet for tablet 8bbf33f0ec4a4adb9a979b08606ee822 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=09a2b90df269425ba3ab97f05a1fc119]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:05.778628 7079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bbf33f0ec4a4adb9a979b08606ee822. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:05.779891 7303 tablet_bootstrap.cc:492] T d6c2626b3ee7493c8a349f08e17082b5 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:46:05.780113 7299 tablet_replica.cc:330] T 48c3fb2ce0fc41a4a12a757052beccb8 P ed47c023452a497192f35d2f617bf9bd: stopping tablet replica
I20240502 20:46:05.780467 7299 raft_consensus.cc:2229] T 48c3fb2ce0fc41a4a12a757052beccb8 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:05.781160 7298 tablet_bootstrap.cc:492] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:46:05.781558 7299 raft_consensus.cc:2258] T 48c3fb2ce0fc41a4a12a757052beccb8 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:05.783023 7162 tablet_bootstrap.cc:492] T bcfd626ea0a04fa7b03dd0ede6b35266 P 7845c397230d49358ba2fad9db53f97c: No bootstrap required, opened a new log
I20240502 20:46:05.783116 7162 ts_tablet_manager.cc:1367] T bcfd626ea0a04fa7b03dd0ede6b35266 P 7845c397230d49358ba2fad9db53f97c: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.000s
I20240502 20:46:05.783634 7162 raft_consensus.cc:348] T bcfd626ea0a04fa7b03dd0ede6b35266 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:05.783744 7162 raft_consensus.cc:374] T bcfd626ea0a04fa7b03dd0ede6b35266 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:05.783775 7162 raft_consensus.cc:729] T bcfd626ea0a04fa7b03dd0ede6b35266 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Initialized, Role: FOLLOWER
I20240502 20:46:05.783898 7162 consensus_queue.cc:260] T bcfd626ea0a04fa7b03dd0ede6b35266 P 7845c397230d49358ba2fad9db53f97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:05.784147 7162 ts_tablet_manager.cc:1398] T bcfd626ea0a04fa7b03dd0ede6b35266 P 7845c397230d49358ba2fad9db53f97c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:05.785511 7304 tablet_bootstrap.cc:492] T 8bbf33f0ec4a4adb9a979b08606ee822 P a47e5c56f13843018ac584914a8d4707: Bootstrap starting.
I20240502 20:46:05.786197 7166 tablet_bootstrap.cc:492] T d6c2626b3ee7493c8a349f08e17082b5 P a4bf1df7fa6b4948be502c182bfbb833: No bootstrap required, opened a new log
I20240502 20:46:05.786305 7166 ts_tablet_manager.cc:1367] T d6c2626b3ee7493c8a349f08e17082b5 P a4bf1df7fa6b4948be502c182bfbb833: Time spent bootstrapping tablet: real 0.027s user 0.000s sys 0.002s
I20240502 20:46:05.786867 7166 raft_consensus.cc:348] T d6c2626b3ee7493c8a349f08e17082b5 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:05.786974 7166 raft_consensus.cc:374] T d6c2626b3ee7493c8a349f08e17082b5 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:05.787010 7166 raft_consensus.cc:729] T d6c2626b3ee7493c8a349f08e17082b5 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Initialized, Role: FOLLOWER
I20240502 20:46:05.787142 7166 consensus_queue.cc:260] T d6c2626b3ee7493c8a349f08e17082b5 P a4bf1df7fa6b4948be502c182bfbb833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:05.787429 7166 ts_tablet_manager.cc:1398] T d6c2626b3ee7493c8a349f08e17082b5 P a4bf1df7fa6b4948be502c182bfbb833: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:05.790668 7305 tablet_bootstrap.cc:492] T 8bbf33f0ec4a4adb9a979b08606ee822 P 201d2b6485084e7dad2cbb722af94779: Bootstrap starting.
I20240502 20:46:05.793188 7305 tablet_bootstrap.cc:654] T 8bbf33f0ec4a4adb9a979b08606ee822 P 201d2b6485084e7dad2cbb722af94779: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:05.793664 7304 tablet_bootstrap.cc:654] T 8bbf33f0ec4a4adb9a979b08606ee822 P a47e5c56f13843018ac584914a8d4707: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:05.794590 7304 log.cc:824] T 8bbf33f0ec4a4adb9a979b08606ee822 P a47e5c56f13843018ac584914a8d4707: Log is configured to *not* fsync() on all Append() calls
I20240502 20:46:05.794587 7305 log.cc:824] T 8bbf33f0ec4a4adb9a979b08606ee822 P 201d2b6485084e7dad2cbb722af94779: Log is configured to *not* fsync() on all Append() calls
I20240502 20:46:05.794719 7298 tablet_bootstrap.cc:654] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:05.795439 7296 tablet_replica.cc:330] T 3cb28ecff16240afb413a1d0e51733b0 P 923cfab926e7443f8ffd7b21ad0a9d91: stopping tablet replica
I20240502 20:46:05.795771 7303 tablet_bootstrap.cc:654] T d6c2626b3ee7493c8a349f08e17082b5 P 923cfab926e7443f8ffd7b21ad0a9d91: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:05.796712 7303 tablet_bootstrap.cc:492] T d6c2626b3ee7493c8a349f08e17082b5 P 923cfab926e7443f8ffd7b21ad0a9d91: No bootstrap required, opened a new log
I20240502 20:46:05.796864 7303 ts_tablet_manager.cc:1367] T d6c2626b3ee7493c8a349f08e17082b5 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240502 20:46:05.796873 7304 tablet_bootstrap.cc:492] T 8bbf33f0ec4a4adb9a979b08606ee822 P a47e5c56f13843018ac584914a8d4707: No bootstrap required, opened a new log
I20240502 20:46:05.796927 7295 tablet_replica.cc:330] T 3cb28ecff16240afb413a1d0e51733b0 P c3d26eafd51547acb780d4aac5ebe7cc: stopping tablet replica
I20240502 20:46:05.797067 7304 ts_tablet_manager.cc:1367] T 8bbf33f0ec4a4adb9a979b08606ee822 P a47e5c56f13843018ac584914a8d4707: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20240502 20:46:05.795773 7296 raft_consensus.cc:2229] T 3cb28ecff16240afb413a1d0e51733b0 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:05.797436 7303 raft_consensus.cc:348] T d6c2626b3ee7493c8a349f08e17082b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:05.797535 7296 raft_consensus.cc:2258] T 3cb28ecff16240afb413a1d0e51733b0 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:05.797626 7303 raft_consensus.cc:374] T d6c2626b3ee7493c8a349f08e17082b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:05.797766 7303 raft_consensus.cc:729] T d6c2626b3ee7493c8a349f08e17082b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: FOLLOWER
I20240502 20:46:05.797935 7303 consensus_queue.cc:260] T d6c2626b3ee7493c8a349f08e17082b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:05.798511 7303 ts_tablet_manager.cc:1398] T d6c2626b3ee7493c8a349f08e17082b5 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:05.799733 7295 raft_consensus.cc:2229] T 3cb28ecff16240afb413a1d0e51733b0 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:05.800637 7296 ts_tablet_manager.cc:1875] T 3cb28ecff16240afb413a1d0e51733b0 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:05.801225 7299 ts_tablet_manager.cc:1875] T 48c3fb2ce0fc41a4a12a757052beccb8 P ed47c023452a497192f35d2f617bf9bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:05.801203 7295 raft_consensus.cc:2258] T 3cb28ecff16240afb413a1d0e51733b0 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:05.803339 4230 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 20:46:05.803553 7295 ts_tablet_manager.cc:1875] T 3cb28ecff16240afb413a1d0e51733b0 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:05.804608 7296 ts_tablet_manager.cc:1888] T 3cb28ecff16240afb413a1d0e51733b0 P 923cfab926e7443f8ffd7b21ad0a9d91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 20:46:05.804716 7296 log.cc:1196] T 3cb28ecff16240afb413a1d0e51733b0 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal/wals/3cb28ecff16240afb413a1d0e51733b0
I20240502 20:46:05.805097 7296 ts_tablet_manager.cc:1909] T 3cb28ecff16240afb413a1d0e51733b0 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting consensus metadata
I20240502 20:46:05.806134 7296 tablet_replica.cc:330] T 84a90777f631447d8ba3ffb720d370e2 P 923cfab926e7443f8ffd7b21ad0a9d91: stopping tablet replica
I20240502 20:46:05.807196 7298 tablet_bootstrap.cc:492] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:46:05.807384 7298 ts_tablet_manager.cc:1367] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.000s
I20240502 20:46:05.807441 4219 catalog_manager.cc:4830] TS 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141): tablet 3cb28ecff16240afb413a1d0e51733b0 (table rebalance_test_table_0 [id=9a093bb257b04f07a867d916725eb265]) successfully deleted
I20240502 20:46:05.808020 7298 raft_consensus.cc:348] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:05.808146 7298 raft_consensus.cc:374] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:05.808192 7298 raft_consensus.cc:729] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:46:05.808342 7298 consensus_queue.cc:260] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:05.808379 7296 raft_consensus.cc:2229] T 84a90777f631447d8ba3ffb720d370e2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:05.808399 7295 ts_tablet_manager.cc:1888] T 3cb28ecff16240afb413a1d0e51733b0 P c3d26eafd51547acb780d4aac5ebe7cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 20:46:05.808477 7295 log.cc:1196] T 3cb28ecff16240afb413a1d0e51733b0 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal/wals/3cb28ecff16240afb413a1d0e51733b0
I20240502 20:46:05.808683 7298 ts_tablet_manager.cc:1398] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:05.808949 7296 raft_consensus.cc:2258] T 84a90777f631447d8ba3ffb720d370e2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:05.809535 7295 ts_tablet_manager.cc:1909] T 3cb28ecff16240afb413a1d0e51733b0 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting consensus metadata
I20240502 20:46:05.810355 7299 ts_tablet_manager.cc:1888] T 48c3fb2ce0fc41a4a12a757052beccb8 P ed47c023452a497192f35d2f617bf9bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 20:46:05.809556 7296 ts_tablet_manager.cc:1875] T 84a90777f631447d8ba3ffb720d370e2 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:05.810482 7299 log.cc:1196] T 48c3fb2ce0fc41a4a12a757052beccb8 P ed47c023452a497192f35d2f617bf9bd: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal/wals/48c3fb2ce0fc41a4a12a757052beccb8
I20240502 20:46:05.810628 7295 tablet_replica.cc:330] T 48c3fb2ce0fc41a4a12a757052beccb8 P c3d26eafd51547acb780d4aac5ebe7cc: stopping tablet replica
I20240502 20:46:05.810811 7299 ts_tablet_manager.cc:1909] T 48c3fb2ce0fc41a4a12a757052beccb8 P ed47c023452a497192f35d2f617bf9bd: Deleting consensus metadata
I20240502 20:46:05.811617 4221 catalog_manager.cc:4830] TS c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097): tablet 3cb28ecff16240afb413a1d0e51733b0 (table rebalance_test_table_0 [id=9a093bb257b04f07a867d916725eb265]) successfully deleted
I20240502 20:46:05.813359 7295 raft_consensus.cc:2229] T 48c3fb2ce0fc41a4a12a757052beccb8 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:05.813498 7295 raft_consensus.cc:2258] T 48c3fb2ce0fc41a4a12a757052beccb8 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:05.814018 7295 ts_tablet_manager.cc:1875] T 48c3fb2ce0fc41a4a12a757052beccb8 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:05.814543 7304 raft_consensus.cc:348] T 8bbf33f0ec4a4adb9a979b08606ee822 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:05.814900 7304 raft_consensus.cc:374] T 8bbf33f0ec4a4adb9a979b08606ee822 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:05.815006 7304 raft_consensus.cc:729] T 8bbf33f0ec4a4adb9a979b08606ee822 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Initialized, Role: FOLLOWER
I20240502 20:46:05.815518 7296 ts_tablet_manager.cc:1888] T 84a90777f631447d8ba3ffb720d370e2 P 923cfab926e7443f8ffd7b21ad0a9d91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 20:46:05.815933 7304 consensus_queue.cc:260] T 8bbf33f0ec4a4adb9a979b08606ee822 P a47e5c56f13843018ac584914a8d4707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:05.816145 7296 log.cc:1196] T 84a90777f631447d8ba3ffb720d370e2 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal/wals/84a90777f631447d8ba3ffb720d370e2
I20240502 20:46:05.816198 7295 ts_tablet_manager.cc:1888] T 48c3fb2ce0fc41a4a12a757052beccb8 P c3d26eafd51547acb780d4aac5ebe7cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 20:46:05.816282 7295 log.cc:1196] T 48c3fb2ce0fc41a4a12a757052beccb8 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal/wals/48c3fb2ce0fc41a4a12a757052beccb8
I20240502 20:46:05.816571 7295 ts_tablet_manager.cc:1909] T 48c3fb2ce0fc41a4a12a757052beccb8 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting consensus metadata
I20240502 20:46:05.816944 7234 tablet.cc:1722] T 125186c6a01347dd96c1b111cbe8b64e P a4bf1df7fa6b4948be502c182bfbb833: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 20:46:05.817270 4221 catalog_manager.cc:4830] TS c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097): tablet 48c3fb2ce0fc41a4a12a757052beccb8 (table rebalance_test_table_0 [id=9a093bb257b04f07a867d916725eb265]) successfully deleted
I20240502 20:46:05.817427 7296 ts_tablet_manager.cc:1909] T 84a90777f631447d8ba3ffb720d370e2 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting consensus metadata
I20240502 20:46:05.818248 4219 catalog_manager.cc:4830] TS 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141): tablet 84a90777f631447d8ba3ffb720d370e2 (table rebalance_test_table_0 [id=9a093bb257b04f07a867d916725eb265]) successfully deleted
I20240502 20:46:05.818419 7304 ts_tablet_manager.cc:1398] T 8bbf33f0ec4a4adb9a979b08606ee822 P a47e5c56f13843018ac584914a8d4707: Time spent starting tablet: real 0.021s user 0.006s sys 0.000s
I20240502 20:46:05.818720 7305 tablet_bootstrap.cc:492] T 8bbf33f0ec4a4adb9a979b08606ee822 P 201d2b6485084e7dad2cbb722af94779: No bootstrap required, opened a new log
I20240502 20:46:05.818831 7013 heartbeater.cc:503] Master 127.30.32.126:40167 was elected leader, sending a full tablet report...
I20240502 20:46:05.818922 7305 ts_tablet_manager.cc:1367] T 8bbf33f0ec4a4adb9a979b08606ee822 P 201d2b6485084e7dad2cbb722af94779: Time spent bootstrapping tablet: real 0.028s user 0.001s sys 0.004s
I20240502 20:46:05.820494 7242 tablet.cc:1722] T 125186c6a01347dd96c1b111cbe8b64e P 7845c397230d49358ba2fad9db53f97c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 20:46:05.821775 7304 tablet_bootstrap.cc:492] T bcfd626ea0a04fa7b03dd0ede6b35266 P a47e5c56f13843018ac584914a8d4707: Bootstrap starting.
I20240502 20:46:05.822762 7304 tablet_bootstrap.cc:654] T bcfd626ea0a04fa7b03dd0ede6b35266 P a47e5c56f13843018ac584914a8d4707: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:05.822824 7305 raft_consensus.cc:348] T 8bbf33f0ec4a4adb9a979b08606ee822 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:05.823050 7305 raft_consensus.cc:374] T 8bbf33f0ec4a4adb9a979b08606ee822 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:05.823107 7305 raft_consensus.cc:729] T 8bbf33f0ec4a4adb9a979b08606ee822 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Initialized, Role: FOLLOWER
I20240502 20:46:05.823170 7017 raft_consensus.cc:482] T d6c2626b3ee7493c8a349f08e17082b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:05.823282 7017 raft_consensus.cc:504] T d6c2626b3ee7493c8a349f08e17082b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:05.823541 7305 consensus_queue.cc:260] T 8bbf33f0ec4a4adb9a979b08606ee822 P 201d2b6485084e7dad2cbb722af94779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:05.823782 7017 leader_election.cc:288] T d6c2626b3ee7493c8a349f08e17082b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867)
I20240502 20:46:05.824090 7310 raft_consensus.cc:482] T 8bbf33f0ec4a4adb9a979b08606ee822 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:05.824220 7310 raft_consensus.cc:504] T 8bbf33f0ec4a4adb9a979b08606ee822 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:05.824412 7305 ts_tablet_manager.cc:1398] T 8bbf33f0ec4a4adb9a979b08606ee822 P 201d2b6485084e7dad2cbb722af94779: Time spent starting tablet: real 0.005s user 0.003s sys 0.001s
I20240502 20:46:05.824826 7145 heartbeater.cc:503] Master 127.30.32.126:40167 was elected leader, sending a full tablet report...
I20240502 20:46:05.826350 7233 tablet.cc:1722] T 125186c6a01347dd96c1b111cbe8b64e P c3d26eafd51547acb780d4aac5ebe7cc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 20:46:05.827049 7310 leader_election.cc:288] T 8bbf33f0ec4a4adb9a979b08606ee822 P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
I20240502 20:46:05.828552 4234 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: rebalancer_dynamic_table_new_name [id=67b3a17c6cdb49808f2f3fde61ef1f54] alter complete (version 3)
I20240502 20:46:05.828598 7305 tablet_bootstrap.cc:492] T bcfd626ea0a04fa7b03dd0ede6b35266 P 201d2b6485084e7dad2cbb722af94779: Bootstrap starting.
I20240502 20:46:05.829766 7305 tablet_bootstrap.cc:654] T bcfd626ea0a04fa7b03dd0ede6b35266 P 201d2b6485084e7dad2cbb722af94779: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:05.830984 7305 tablet_bootstrap.cc:492] T bcfd626ea0a04fa7b03dd0ede6b35266 P 201d2b6485084e7dad2cbb722af94779: No bootstrap required, opened a new log
I20240502 20:46:05.831090 7305 ts_tablet_manager.cc:1367] T bcfd626ea0a04fa7b03dd0ede6b35266 P 201d2b6485084e7dad2cbb722af94779: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:46:05.831697 7305 raft_consensus.cc:348] T bcfd626ea0a04fa7b03dd0ede6b35266 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:05.832063 7305 raft_consensus.cc:374] T bcfd626ea0a04fa7b03dd0ede6b35266 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:05.832111 7305 raft_consensus.cc:729] T bcfd626ea0a04fa7b03dd0ede6b35266 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Initialized, Role: FOLLOWER
I20240502 20:46:05.832270 7299 tablet_replica.cc:330] T 84a90777f631447d8ba3ffb720d370e2 P ed47c023452a497192f35d2f617bf9bd: stopping tablet replica
I20240502 20:46:05.832304 7305 consensus_queue.cc:260] T bcfd626ea0a04fa7b03dd0ede6b35266 P 201d2b6485084e7dad2cbb722af94779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:05.832461 7299 raft_consensus.cc:2229] T 84a90777f631447d8ba3ffb720d370e2 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:05.832651 7299 raft_consensus.cc:2258] T 84a90777f631447d8ba3ffb720d370e2 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:05.832829 7305 ts_tablet_manager.cc:1398] T bcfd626ea0a04fa7b03dd0ede6b35266 P 201d2b6485084e7dad2cbb722af94779: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:05.833271 4222 catalog_manager.cc:4830] TS ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193): tablet 48c3fb2ce0fc41a4a12a757052beccb8 (table rebalance_test_table_0 [id=9a093bb257b04f07a867d916725eb265]) successfully deleted
I20240502 20:46:05.833330 7299 ts_tablet_manager.cc:1875] T 84a90777f631447d8ba3ffb720d370e2 P ed47c023452a497192f35d2f617bf9bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:05.835872 4374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bbf33f0ec4a4adb9a979b08606ee822" candidate_uuid: "a47e5c56f13843018ac584914a8d4707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2" is_pre_election: true
I20240502 20:46:05.836019 4374 raft_consensus.cc:2454] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a47e5c56f13843018ac584914a8d4707 in term 0.
I20240502 20:46:05.836391 6903 leader_election.cc:302] T 8bbf33f0ec4a4adb9a979b08606ee822 P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a47e5c56f13843018ac584914a8d4707, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:46:05.837802 7304 tablet_bootstrap.cc:492] T bcfd626ea0a04fa7b03dd0ede6b35266 P a47e5c56f13843018ac584914a8d4707: No bootstrap required, opened a new log
I20240502 20:46:05.837893 7304 ts_tablet_manager.cc:1367] T bcfd626ea0a04fa7b03dd0ede6b35266 P a47e5c56f13843018ac584914a8d4707: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240502 20:46:05.838454 7304 raft_consensus.cc:348] T bcfd626ea0a04fa7b03dd0ede6b35266 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:05.838562 7304 raft_consensus.cc:374] T bcfd626ea0a04fa7b03dd0ede6b35266 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:05.838593 7304 raft_consensus.cc:729] T bcfd626ea0a04fa7b03dd0ede6b35266 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Initialized, Role: FOLLOWER
I20240502 20:46:05.838719 7304 consensus_queue.cc:260] T bcfd626ea0a04fa7b03dd0ede6b35266 P a47e5c56f13843018ac584914a8d4707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:05.838985 7304 ts_tablet_manager.cc:1398] T bcfd626ea0a04fa7b03dd0ede6b35266 P a47e5c56f13843018ac584914a8d4707: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:05.839165 7304 tablet_bootstrap.cc:492] T d6c2626b3ee7493c8a349f08e17082b5 P a47e5c56f13843018ac584914a8d4707: Bootstrap starting.
I20240502 20:46:05.840003 7304 tablet_bootstrap.cc:654] T d6c2626b3ee7493c8a349f08e17082b5 P a47e5c56f13843018ac584914a8d4707: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:05.840085 6835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2626b3ee7493c8a349f08e17082b5" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833" is_pre_election: true
I20240502 20:46:05.840297 6835 raft_consensus.cc:2454] T d6c2626b3ee7493c8a349f08e17082b5 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 0.
I20240502 20:46:05.840458 7299 ts_tablet_manager.cc:1888] T 84a90777f631447d8ba3ffb720d370e2 P ed47c023452a497192f35d2f617bf9bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 20:46:05.840524 7299 log.cc:1196] T 84a90777f631447d8ba3ffb720d370e2 P ed47c023452a497192f35d2f617bf9bd: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal/wals/84a90777f631447d8ba3ffb720d370e2
I20240502 20:46:05.840663 4573 leader_election.cc:302] T d6c2626b3ee7493c8a349f08e17082b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, a4bf1df7fa6b4948be502c182bfbb833; no voters:
I20240502 20:46:05.840811 7299 ts_tablet_manager.cc:1909] T 84a90777f631447d8ba3ffb720d370e2 P ed47c023452a497192f35d2f617bf9bd: Deleting consensus metadata
I20240502 20:46:05.840893 7017 raft_consensus.cc:2781] T d6c2626b3ee7493c8a349f08e17082b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:05.840965 7017 raft_consensus.cc:482] T d6c2626b3ee7493c8a349f08e17082b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:05.841028 7017 raft_consensus.cc:3037] T d6c2626b3ee7493c8a349f08e17082b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:05.841126 7304 tablet_bootstrap.cc:492] T d6c2626b3ee7493c8a349f08e17082b5 P a47e5c56f13843018ac584914a8d4707: No bootstrap required, opened a new log
I20240502 20:46:05.841199 7304 ts_tablet_manager.cc:1367] T d6c2626b3ee7493c8a349f08e17082b5 P a47e5c56f13843018ac584914a8d4707: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:05.841509 4222 catalog_manager.cc:4830] TS ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193): tablet 84a90777f631447d8ba3ffb720d370e2 (table rebalance_test_table_0 [id=9a093bb257b04f07a867d916725eb265]) successfully deleted
I20240502 20:46:05.841578 7304 raft_consensus.cc:348] T d6c2626b3ee7493c8a349f08e17082b5 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:05.841662 7304 raft_consensus.cc:374] T d6c2626b3ee7493c8a349f08e17082b5 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:05.841689 7304 raft_consensus.cc:729] T d6c2626b3ee7493c8a349f08e17082b5 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Initialized, Role: FOLLOWER
I20240502 20:46:05.841791 7304 consensus_queue.cc:260] T d6c2626b3ee7493c8a349f08e17082b5 P a47e5c56f13843018ac584914a8d4707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:05.841956 7017 raft_consensus.cc:504] T d6c2626b3ee7493c8a349f08e17082b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:05.842109 7304 ts_tablet_manager.cc:1398] T d6c2626b3ee7493c8a349f08e17082b5 P a47e5c56f13843018ac584914a8d4707: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:05.842136 7322 raft_consensus.cc:2781] T 8bbf33f0ec4a4adb9a979b08606ee822 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:05.842204 7322 raft_consensus.cc:482] T 8bbf33f0ec4a4adb9a979b08606ee822 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:05.842253 7322 raft_consensus.cc:3037] T 8bbf33f0ec4a4adb9a979b08606ee822 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:05.842382 7017 leader_election.cc:288] T d6c2626b3ee7493c8a349f08e17082b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 election: Requested vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867)
I20240502 20:46:05.843506 7322 raft_consensus.cc:504] T 8bbf33f0ec4a4adb9a979b08606ee822 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:05.843830 7322 leader_election.cc:288] T 8bbf33f0ec4a4adb9a979b08606ee822 P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 1 election: Requested vote from peers c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
I20240502 20:46:05.845597 4374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bbf33f0ec4a4adb9a979b08606ee822" candidate_uuid: "a47e5c56f13843018ac584914a8d4707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2"
I20240502 20:46:05.845742 4374 raft_consensus.cc:3037] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:05.846125 6835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2626b3ee7493c8a349f08e17082b5" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833"
I20240502 20:46:05.846240 6835 raft_consensus.cc:3037] T d6c2626b3ee7493c8a349f08e17082b5 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:05.846745 4374 raft_consensus.cc:2454] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a47e5c56f13843018ac584914a8d4707 in term 1.
I20240502 20:46:05.847072 6903 leader_election.cc:302] T 8bbf33f0ec4a4adb9a979b08606ee822 P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a47e5c56f13843018ac584914a8d4707, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:46:05.847240 7322 raft_consensus.cc:2781] T 8bbf33f0ec4a4adb9a979b08606ee822 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:05.847252 6835 raft_consensus.cc:2454] T d6c2626b3ee7493c8a349f08e17082b5 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 1.
I20240502 20:46:05.847342 7322 raft_consensus.cc:686] T 8bbf33f0ec4a4adb9a979b08606ee822 P a47e5c56f13843018ac584914a8d4707 [term 1 LEADER]: Becoming Leader. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Running, Role: LEADER
I20240502 20:46:05.847499 4573 leader_election.cc:302] T d6c2626b3ee7493c8a349f08e17082b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, a4bf1df7fa6b4948be502c182bfbb833; no voters:
I20240502 20:46:05.847664 7017 raft_consensus.cc:2781] T d6c2626b3ee7493c8a349f08e17082b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:05.847743 7017 raft_consensus.cc:686] T d6c2626b3ee7493c8a349f08e17082b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Becoming Leader. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Running, Role: LEADER
I20240502 20:46:05.847857 7017 consensus_queue.cc:237] T d6c2626b3ee7493c8a349f08e17082b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:05.849021 6957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2626b3ee7493c8a349f08e17082b5" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47e5c56f13843018ac584914a8d4707" is_pre_election: true
I20240502 20:46:05.849203 6957 raft_consensus.cc:2454] T d6c2626b3ee7493c8a349f08e17082b5 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 0.
I20240502 20:46:05.849465 6967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2626b3ee7493c8a349f08e17082b5" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47e5c56f13843018ac584914a8d4707"
I20240502 20:46:05.849553 6967 raft_consensus.cc:3037] T d6c2626b3ee7493c8a349f08e17082b5 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:05.849972 7322 consensus_queue.cc:237] T 8bbf33f0ec4a4adb9a979b08606ee822 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:05.850471 6967 raft_consensus.cc:2454] T d6c2626b3ee7493c8a349f08e17082b5 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 1.
I20240502 20:46:05.851563 4235 catalog_manager.cc:5484] T d6c2626b3ee7493c8a349f08e17082b5 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67). New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } }
I20240502 20:46:05.854138 7099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bbf33f0ec4a4adb9a979b08606ee822" candidate_uuid: "a47e5c56f13843018ac584914a8d4707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201d2b6485084e7dad2cbb722af94779"
I20240502 20:46:05.854430 7099 raft_consensus.cc:3037] T 8bbf33f0ec4a4adb9a979b08606ee822 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:05.854138 7098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bbf33f0ec4a4adb9a979b08606ee822" candidate_uuid: "a47e5c56f13843018ac584914a8d4707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201d2b6485084e7dad2cbb722af94779" is_pre_election: true
I20240502 20:46:05.855610 7099 raft_consensus.cc:2454] T 8bbf33f0ec4a4adb9a979b08606ee822 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a47e5c56f13843018ac584914a8d4707 in term 1.
I20240502 20:46:05.865561 4235 catalog_manager.cc:5484] T 8bbf33f0ec4a4adb9a979b08606ee822 P a47e5c56f13843018ac584914a8d4707 reported cstate change: term changed from 0 to 1, leader changed from <none> to a47e5c56f13843018ac584914a8d4707 (127.30.32.71). New cstate: current_term: 1 leader_uuid: "a47e5c56f13843018ac584914a8d4707" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:05.935704 4232 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 20:46:05.935837 4232 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 20:46:05.937899 4232 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: Sending DeleteTablet for 3 replicas of tablet 125186c6a01347dd96c1b111cbe8b64e
I20240502 20:46:05.938522 6809 tablet_service.cc:1514] Processing DeleteTablet for tablet 125186c6a01347dd96c1b111cbe8b64e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:05 UTC) from {username='slave'} at 127.0.0.1:35290
I20240502 20:46:05.938874 6683 tablet_service.cc:1514] Processing DeleteTablet for tablet 125186c6a01347dd96c1b111cbe8b64e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:05 UTC) from {username='slave'} at 127.0.0.1:37506
I20240502 20:46:05.939116 4488 tablet_service.cc:1514] Processing DeleteTablet for tablet 125186c6a01347dd96c1b111cbe8b64e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:05 UTC) from {username='slave'} at 127.0.0.1:48890
I20240502 20:46:05.939316 7295 tablet_replica.cc:330] T 125186c6a01347dd96c1b111cbe8b64e P c3d26eafd51547acb780d4aac5ebe7cc: stopping tablet replica
I20240502 20:46:05.939435 7366 tablet_replica.cc:330] T 125186c6a01347dd96c1b111cbe8b64e P a4bf1df7fa6b4948be502c182bfbb833: stopping tablet replica
I20240502 20:46:05.939481 7295 raft_consensus.cc:2229] T 125186c6a01347dd96c1b111cbe8b64e P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:05.939611 7295 raft_consensus.cc:2258] T 125186c6a01347dd96c1b111cbe8b64e P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:05.940313 7295 ts_tablet_manager.cc:1875] T 125186c6a01347dd96c1b111cbe8b64e P c3d26eafd51547acb780d4aac5ebe7cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:05.940959 7367 tablet_replica.cc:330] T 125186c6a01347dd96c1b111cbe8b64e P 7845c397230d49358ba2fad9db53f97c: stopping tablet replica
I20240502 20:46:05.941258 7367 raft_consensus.cc:2229] T 125186c6a01347dd96c1b111cbe8b64e P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:05.941524 7367 raft_consensus.cc:2258] T 125186c6a01347dd96c1b111cbe8b64e P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:05.942651 7295 ts_tablet_manager.cc:1888] T 125186c6a01347dd96c1b111cbe8b64e P c3d26eafd51547acb780d4aac5ebe7cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:05.942690 7367 ts_tablet_manager.cc:1875] T 125186c6a01347dd96c1b111cbe8b64e P 7845c397230d49358ba2fad9db53f97c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:05.942857 7295 log.cc:1196] T 125186c6a01347dd96c1b111cbe8b64e P c3d26eafd51547acb780d4aac5ebe7cc: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal/wals/125186c6a01347dd96c1b111cbe8b64e
I20240502 20:46:05.943159 7295 ts_tablet_manager.cc:1909] T 125186c6a01347dd96c1b111cbe8b64e P c3d26eafd51547acb780d4aac5ebe7cc: Deleting consensus metadata
I20240502 20:46:05.943859 7366 raft_consensus.cc:2229] T 125186c6a01347dd96c1b111cbe8b64e P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:05.943913 4221 catalog_manager.cc:4830] TS c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097): tablet 125186c6a01347dd96c1b111cbe8b64e (table rebalancer_dynamic_table_new_name [id=67b3a17c6cdb49808f2f3fde61ef1f54]) successfully deleted
I20240502 20:46:05.944269 7366 raft_consensus.cc:2258] T 125186c6a01347dd96c1b111cbe8b64e P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:05.945629 7367 ts_tablet_manager.cc:1888] T 125186c6a01347dd96c1b111cbe8b64e P 7845c397230d49358ba2fad9db53f97c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:05.945680 7366 ts_tablet_manager.cc:1875] T 125186c6a01347dd96c1b111cbe8b64e P a4bf1df7fa6b4948be502c182bfbb833: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:05.945753 7367 log.cc:1196] T 125186c6a01347dd96c1b111cbe8b64e P 7845c397230d49358ba2fad9db53f97c: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/wal/wals/125186c6a01347dd96c1b111cbe8b64e
I20240502 20:46:05.946108 7367 ts_tablet_manager.cc:1909] T 125186c6a01347dd96c1b111cbe8b64e P 7845c397230d49358ba2fad9db53f97c: Deleting consensus metadata
I20240502 20:46:05.949219 7366 ts_tablet_manager.cc:1888] T 125186c6a01347dd96c1b111cbe8b64e P a4bf1df7fa6b4948be502c182bfbb833: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:05.949309 7366 log.cc:1196] T 125186c6a01347dd96c1b111cbe8b64e P a4bf1df7fa6b4948be502c182bfbb833: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/wal/wals/125186c6a01347dd96c1b111cbe8b64e
I20240502 20:46:05.950084 7366 ts_tablet_manager.cc:1909] T 125186c6a01347dd96c1b111cbe8b64e P a4bf1df7fa6b4948be502c182bfbb833: Deleting consensus metadata
I20240502 20:46:05.951200 4221 catalog_manager.cc:4830] TS 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): tablet 125186c6a01347dd96c1b111cbe8b64e (table rebalancer_dynamic_table_new_name [id=67b3a17c6cdb49808f2f3fde61ef1f54]) successfully deleted
I20240502 20:46:05.951334 4220 catalog_manager.cc:4830] TS a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867): tablet 125186c6a01347dd96c1b111cbe8b64e (table rebalancer_dynamic_table_new_name [id=67b3a17c6cdb49808f2f3fde61ef1f54]) successfully deleted
W20240502 20:46:05.981649 7146 tablet.cc:2369] T 8bbf33f0ec4a4adb9a979b08606ee822 P 201d2b6485084e7dad2cbb722af94779: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 20:46:06.039410 4233 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41812:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 20:46:06.044525 6946 tablet_service.cc:1467] Processing CreateTablet for tablet 2a9efc56d59c444ea3d3338499e7a91e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c3e6475692944a73a98468ce42b4c70a]), partition=
I20240502 20:46:06.044726 6683 tablet_service.cc:1467] Processing CreateTablet for tablet 2a9efc56d59c444ea3d3338499e7a91e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c3e6475692944a73a98468ce42b4c70a]), partition=
I20240502 20:46:06.044752 6809 tablet_service.cc:1467] Processing CreateTablet for tablet 2a9efc56d59c444ea3d3338499e7a91e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c3e6475692944a73a98468ce42b4c70a]), partition=
I20240502 20:46:06.044839 6946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a9efc56d59c444ea3d3338499e7a91e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:06.044998 6809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a9efc56d59c444ea3d3338499e7a91e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:06.045011 6683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a9efc56d59c444ea3d3338499e7a91e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:06.046622 7162 tablet_bootstrap.cc:492] T 2a9efc56d59c444ea3d3338499e7a91e P 7845c397230d49358ba2fad9db53f97c: Bootstrap starting.
I20240502 20:46:06.046947 7166 tablet_bootstrap.cc:492] T 2a9efc56d59c444ea3d3338499e7a91e P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap starting.
I20240502 20:46:06.047549 7162 tablet_bootstrap.cc:654] T 2a9efc56d59c444ea3d3338499e7a91e P 7845c397230d49358ba2fad9db53f97c: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:06.047783 7304 tablet_bootstrap.cc:492] T 2a9efc56d59c444ea3d3338499e7a91e P a47e5c56f13843018ac584914a8d4707: Bootstrap starting.
I20240502 20:46:06.047856 7166 tablet_bootstrap.cc:654] T 2a9efc56d59c444ea3d3338499e7a91e P a4bf1df7fa6b4948be502c182bfbb833: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:06.048718 7304 tablet_bootstrap.cc:654] T 2a9efc56d59c444ea3d3338499e7a91e P a47e5c56f13843018ac584914a8d4707: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:06.048774 7162 tablet_bootstrap.cc:492] T 2a9efc56d59c444ea3d3338499e7a91e P 7845c397230d49358ba2fad9db53f97c: No bootstrap required, opened a new log
I20240502 20:46:06.048853 7162 ts_tablet_manager.cc:1367] T 2a9efc56d59c444ea3d3338499e7a91e P 7845c397230d49358ba2fad9db53f97c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:06.048938 7166 tablet_bootstrap.cc:492] T 2a9efc56d59c444ea3d3338499e7a91e P a4bf1df7fa6b4948be502c182bfbb833: No bootstrap required, opened a new log
I20240502 20:46:06.049041 7166 ts_tablet_manager.cc:1367] T 2a9efc56d59c444ea3d3338499e7a91e P a4bf1df7fa6b4948be502c182bfbb833: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 20:46:06.049336 7162 raft_consensus.cc:348] T 2a9efc56d59c444ea3d3338499e7a91e P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:06.049448 7162 raft_consensus.cc:374] T 2a9efc56d59c444ea3d3338499e7a91e P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:06.049494 7162 raft_consensus.cc:729] T 2a9efc56d59c444ea3d3338499e7a91e P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Initialized, Role: FOLLOWER
I20240502 20:46:06.049533 7166 raft_consensus.cc:348] T 2a9efc56d59c444ea3d3338499e7a91e P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:06.049641 7166 raft_consensus.cc:374] T 2a9efc56d59c444ea3d3338499e7a91e P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:06.049675 7166 raft_consensus.cc:729] T 2a9efc56d59c444ea3d3338499e7a91e P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Initialized, Role: FOLLOWER
I20240502 20:46:06.049639 7162 consensus_queue.cc:260] T 2a9efc56d59c444ea3d3338499e7a91e P 7845c397230d49358ba2fad9db53f97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:06.049793 7166 consensus_queue.cc:260] T 2a9efc56d59c444ea3d3338499e7a91e P a4bf1df7fa6b4948be502c182bfbb833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:06.050057 7304 tablet_bootstrap.cc:492] T 2a9efc56d59c444ea3d3338499e7a91e P a47e5c56f13843018ac584914a8d4707: No bootstrap required, opened a new log
I20240502 20:46:06.050065 7166 ts_tablet_manager.cc:1398] T 2a9efc56d59c444ea3d3338499e7a91e P a4bf1df7fa6b4948be502c182bfbb833: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:06.050099 7162 ts_tablet_manager.cc:1398] T 2a9efc56d59c444ea3d3338499e7a91e P 7845c397230d49358ba2fad9db53f97c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:06.050143 7304 ts_tablet_manager.cc:1367] T 2a9efc56d59c444ea3d3338499e7a91e P a47e5c56f13843018ac584914a8d4707: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 20:46:06.050546 7304 raft_consensus.cc:348] T 2a9efc56d59c444ea3d3338499e7a91e P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:06.050644 7304 raft_consensus.cc:374] T 2a9efc56d59c444ea3d3338499e7a91e P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:06.050673 7304 raft_consensus.cc:729] T 2a9efc56d59c444ea3d3338499e7a91e P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Initialized, Role: FOLLOWER
I20240502 20:46:06.050796 7304 consensus_queue.cc:260] T 2a9efc56d59c444ea3d3338499e7a91e P a47e5c56f13843018ac584914a8d4707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:06.051049 7304 ts_tablet_manager.cc:1398] T 2a9efc56d59c444ea3d3338499e7a91e P a47e5c56f13843018ac584914a8d4707: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240502 20:46:06.052356 7014 tablet.cc:2369] T 2a9efc56d59c444ea3d3338499e7a91e P a47e5c56f13843018ac584914a8d4707: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 20:46:06.098421 7322 raft_consensus.cc:482] T bcfd626ea0a04fa7b03dd0ede6b35266 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:06.098560 7322 raft_consensus.cc:504] T bcfd626ea0a04fa7b03dd0ede6b35266 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:06.101449 7322 leader_election.cc:288] T bcfd626ea0a04fa7b03dd0ede6b35266 P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
I20240502 20:46:06.102514 7099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcfd626ea0a04fa7b03dd0ede6b35266" candidate_uuid: "a47e5c56f13843018ac584914a8d4707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201d2b6485084e7dad2cbb722af94779" is_pre_election: true
I20240502 20:46:06.102674 7099 raft_consensus.cc:2454] T bcfd626ea0a04fa7b03dd0ede6b35266 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a47e5c56f13843018ac584914a8d4707 in term 0.
I20240502 20:46:06.102972 6900 leader_election.cc:302] T bcfd626ea0a04fa7b03dd0ede6b35266 P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201d2b6485084e7dad2cbb722af94779, a47e5c56f13843018ac584914a8d4707; no voters:
I20240502 20:46:06.103192 7322 raft_consensus.cc:2781] T bcfd626ea0a04fa7b03dd0ede6b35266 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:06.103258 7322 raft_consensus.cc:482] T bcfd626ea0a04fa7b03dd0ede6b35266 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:06.103296 7322 raft_consensus.cc:3037] T bcfd626ea0a04fa7b03dd0ede6b35266 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:06.104277 7322 raft_consensus.cc:504] T bcfd626ea0a04fa7b03dd0ede6b35266 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:06.104638 7322 leader_election.cc:288] T bcfd626ea0a04fa7b03dd0ede6b35266 P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 1 election: Requested vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
I20240502 20:46:06.105006 7099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcfd626ea0a04fa7b03dd0ede6b35266" candidate_uuid: "a47e5c56f13843018ac584914a8d4707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201d2b6485084e7dad2cbb722af94779"
I20240502 20:46:06.105131 7099 raft_consensus.cc:3037] T bcfd626ea0a04fa7b03dd0ede6b35266 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:06.105863 6703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcfd626ea0a04fa7b03dd0ede6b35266" candidate_uuid: "a47e5c56f13843018ac584914a8d4707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7845c397230d49358ba2fad9db53f97c" is_pre_election: true
I20240502 20:46:06.105952 6702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcfd626ea0a04fa7b03dd0ede6b35266" candidate_uuid: "a47e5c56f13843018ac584914a8d4707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7845c397230d49358ba2fad9db53f97c"
I20240502 20:46:06.105995 6703 raft_consensus.cc:2454] T bcfd626ea0a04fa7b03dd0ede6b35266 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a47e5c56f13843018ac584914a8d4707 in term 0.
I20240502 20:46:06.106168 7099 raft_consensus.cc:2454] T bcfd626ea0a04fa7b03dd0ede6b35266 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a47e5c56f13843018ac584914a8d4707 in term 1.
I20240502 20:46:06.106523 6900 leader_election.cc:302] T bcfd626ea0a04fa7b03dd0ede6b35266 P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 201d2b6485084e7dad2cbb722af94779, a47e5c56f13843018ac584914a8d4707; no voters: 7845c397230d49358ba2fad9db53f97c
I20240502 20:46:06.106801 7322 raft_consensus.cc:2781] T bcfd626ea0a04fa7b03dd0ede6b35266 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:06.106896 7322 raft_consensus.cc:686] T bcfd626ea0a04fa7b03dd0ede6b35266 P a47e5c56f13843018ac584914a8d4707 [term 1 LEADER]: Becoming Leader. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Running, Role: LEADER
I20240502 20:46:06.107023 7322 consensus_queue.cc:237] T bcfd626ea0a04fa7b03dd0ede6b35266 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:06.108862 4230 catalog_manager.cc:5484] T bcfd626ea0a04fa7b03dd0ede6b35266 P a47e5c56f13843018ac584914a8d4707 reported cstate change: term changed from 0 to 1, leader changed from <none> to a47e5c56f13843018ac584914a8d4707 (127.30.32.71). New cstate: current_term: 1 leader_uuid: "a47e5c56f13843018ac584914a8d4707" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:06.116582 7168 raft_consensus.cc:482] T 2a9efc56d59c444ea3d3338499e7a91e P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:06.116731 7168 raft_consensus.cc:504] T 2a9efc56d59c444ea3d3338499e7a91e P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:06.117324 7168 leader_election.cc:288] T 2a9efc56d59c444ea3d3338499e7a91e P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
I20240502 20:46:06.118089 6702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a9efc56d59c444ea3d3338499e7a91e" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7845c397230d49358ba2fad9db53f97c" is_pre_election: true
I20240502 20:46:06.118257 6702 raft_consensus.cc:2454] T 2a9efc56d59c444ea3d3338499e7a91e P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 0.
I20240502 20:46:06.118665 6770 leader_election.cc:302] T 2a9efc56d59c444ea3d3338499e7a91e P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7845c397230d49358ba2fad9db53f97c, a4bf1df7fa6b4948be502c182bfbb833; no voters:
I20240502 20:46:06.119485 7168 raft_consensus.cc:2781] T 2a9efc56d59c444ea3d3338499e7a91e P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:06.119553 7168 raft_consensus.cc:482] T 2a9efc56d59c444ea3d3338499e7a91e P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:06.119601 7168 raft_consensus.cc:3037] T 2a9efc56d59c444ea3d3338499e7a91e P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:06.120790 7168 raft_consensus.cc:504] T 2a9efc56d59c444ea3d3338499e7a91e P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:06.121217 7168 leader_election.cc:288] T 2a9efc56d59c444ea3d3338499e7a91e P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 election: Requested vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
I20240502 20:46:06.121635 6702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a9efc56d59c444ea3d3338499e7a91e" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7845c397230d49358ba2fad9db53f97c"
I20240502 20:46:06.121666 6967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a9efc56d59c444ea3d3338499e7a91e" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47e5c56f13843018ac584914a8d4707" is_pre_election: true
I20240502 20:46:06.121767 6702 raft_consensus.cc:3037] T 2a9efc56d59c444ea3d3338499e7a91e P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:06.121742 6957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a9efc56d59c444ea3d3338499e7a91e" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47e5c56f13843018ac584914a8d4707"
I20240502 20:46:06.121806 6967 raft_consensus.cc:2454] T 2a9efc56d59c444ea3d3338499e7a91e P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 0.
I20240502 20:46:06.122838 6702 raft_consensus.cc:2454] T 2a9efc56d59c444ea3d3338499e7a91e P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 1.
I20240502 20:46:06.123299 6770 leader_election.cc:302] T 2a9efc56d59c444ea3d3338499e7a91e P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7845c397230d49358ba2fad9db53f97c, a4bf1df7fa6b4948be502c182bfbb833; no voters: a47e5c56f13843018ac584914a8d4707
I20240502 20:46:06.123482 7168 raft_consensus.cc:2781] T 2a9efc56d59c444ea3d3338499e7a91e P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:06.123576 7168 raft_consensus.cc:686] T 2a9efc56d59c444ea3d3338499e7a91e P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEADER]: Becoming Leader. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Running, Role: LEADER
I20240502 20:46:06.123721 7168 consensus_queue.cc:237] T 2a9efc56d59c444ea3d3338499e7a91e P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:06.125593 4230 catalog_manager.cc:5484] T 2a9efc56d59c444ea3d3338499e7a91e P a4bf1df7fa6b4948be502c182bfbb833 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70). New cstate: current_term: 1 leader_uuid: "a4bf1df7fa6b4948be502c182bfbb833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:06.137137 6683 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:06.137229 7078 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:06.141993 6809 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:06.142586 6946 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:46:06.153409 4618 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 20:46:06.165608 4488 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 20:46:06.167562 4749 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 20:46:06.172515 4354 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 20:46:06.238389 4230 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 20:46:06.243057 6957 raft_consensus.cc:3037] T 2a9efc56d59c444ea3d3338499e7a91e P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:06.244390 6957 raft_consensus.cc:1261] T 2a9efc56d59c444ea3d3338499e7a91e P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Refusing update from remote peer a4bf1df7fa6b4948be502c182bfbb833: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:06.245395 6702 raft_consensus.cc:1261] T 2a9efc56d59c444ea3d3338499e7a91e P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Refusing update from remote peer a4bf1df7fa6b4948be502c182bfbb833: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:06.245630 7186 consensus_queue.cc:1035] T 2a9efc56d59c444ea3d3338499e7a91e P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:06.246261 7186 consensus_queue.cc:1035] T 2a9efc56d59c444ea3d3338499e7a91e P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:06.250721 4229 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240502 20:46:06.250967 4229 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240502 20:46:06.251099 7234 tablet.cc:1722] T 2a9efc56d59c444ea3d3338499e7a91e P a4bf1df7fa6b4948be502c182bfbb833: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 20:46:06.251444 7234 tablet.cc:1620] T 2a9efc56d59c444ea3d3338499e7a91e P a4bf1df7fa6b4948be502c182bfbb833: MemRowSet was empty: no flush needed.
I20240502 20:46:06.252048 4233 catalog_manager.cc:5576] TS a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867) does not have the latest schema for tablet 2a9efc56d59c444ea3d3338499e7a91e (table rebalancer_dynamic_table [id=c3e6475692944a73a98468ce42b4c70a]). Expected version 1 got 0
I20240502 20:46:06.252171 7242 tablet.cc:1722] T 2a9efc56d59c444ea3d3338499e7a91e P 7845c397230d49358ba2fad9db53f97c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 20:46:06.252441 7242 tablet.cc:1620] T 2a9efc56d59c444ea3d3338499e7a91e P 7845c397230d49358ba2fad9db53f97c: MemRowSet was empty: no flush needed.
I20240502 20:46:06.252813 4235 catalog_manager.cc:5576] TS a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841) does not have the latest schema for tablet 2a9efc56d59c444ea3d3338499e7a91e (table rebalancer_dynamic_table [id=c3e6475692944a73a98468ce42b4c70a]). Expected version 1 got 0
I20240502 20:46:06.254195 4230 catalog_manager.cc:5576] TS 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429) does not have the latest schema for tablet 2a9efc56d59c444ea3d3338499e7a91e (table rebalancer_dynamic_table [id=c3e6475692944a73a98468ce42b4c70a]). Expected version 1 got 0
I20240502 20:46:06.254504 4229 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: Sending DeleteTablet for 3 replicas of tablet 371e10ffd8e64a33bbd950a140c7c514
I20240502 20:46:06.254796 4229 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: Sending DeleteTablet for 3 replicas of tablet 85e3a77b9daf4f3f93a10b223b168064
I20240502 20:46:06.255041 4229 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: Sending DeleteTablet for 3 replicas of tablet bdac13ec12c24d28abcca7f1e451451d
I20240502 20:46:06.255050 4749 tablet_service.cc:1514] Processing DeleteTablet for tablet 371e10ffd8e64a33bbd950a140c7c514 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:06 UTC) from {username='slave'} at 127.0.0.1:38526
I20240502 20:46:06.255326 7299 tablet_replica.cc:330] T 371e10ffd8e64a33bbd950a140c7c514 P ed47c023452a497192f35d2f617bf9bd: stopping tablet replica
I20240502 20:46:06.255409 4354 tablet_service.cc:1514] Processing DeleteTablet for tablet 371e10ffd8e64a33bbd950a140c7c514 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:06 UTC) from {username='slave'} at 127.0.0.1:38306
I20240502 20:46:06.255494 7299 raft_consensus.cc:2229] T 371e10ffd8e64a33bbd950a140c7c514 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:06.255597 7297 tablet_replica.cc:330] T 371e10ffd8e64a33bbd950a140c7c514 P c92e6e46782849479c8a0c6d82276fb2: stopping tablet replica
I20240502 20:46:06.255661 7299 raft_consensus.cc:2258] T 371e10ffd8e64a33bbd950a140c7c514 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:06.255760 7297 raft_consensus.cc:2229] T 371e10ffd8e64a33bbd950a140c7c514 P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:06.256057 7297 raft_consensus.cc:2258] T 371e10ffd8e64a33bbd950a140c7c514 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:06.256199 7299 ts_tablet_manager.cc:1875] T 371e10ffd8e64a33bbd950a140c7c514 P ed47c023452a497192f35d2f617bf9bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:06.256512 4354 tablet_service.cc:1514] Processing DeleteTablet for tablet 85e3a77b9daf4f3f93a10b223b168064 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:06 UTC) from {username='slave'} at 127.0.0.1:38306
I20240502 20:46:06.256532 4487 tablet_service.cc:1514] Processing DeleteTablet for tablet bdac13ec12c24d28abcca7f1e451451d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:06 UTC) from {username='slave'} at 127.0.0.1:48890
I20240502 20:46:06.256587 7297 ts_tablet_manager.cc:1875] T 371e10ffd8e64a33bbd950a140c7c514 P c92e6e46782849479c8a0c6d82276fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:06.258420 7299 ts_tablet_manager.cc:1888] T 371e10ffd8e64a33bbd950a140c7c514 P ed47c023452a497192f35d2f617bf9bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 20:46:06.258487 7299 log.cc:1196] T 371e10ffd8e64a33bbd950a140c7c514 P ed47c023452a497192f35d2f617bf9bd: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal/wals/371e10ffd8e64a33bbd950a140c7c514
I20240502 20:46:06.258749 7299 ts_tablet_manager.cc:1909] T 371e10ffd8e64a33bbd950a140c7c514 P ed47c023452a497192f35d2f617bf9bd: Deleting consensus metadata
I20240502 20:46:06.258777 7297 ts_tablet_manager.cc:1888] T 371e10ffd8e64a33bbd950a140c7c514 P c92e6e46782849479c8a0c6d82276fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 20:46:06.258847 7297 log.cc:1196] T 371e10ffd8e64a33bbd950a140c7c514 P c92e6e46782849479c8a0c6d82276fb2: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal/wals/371e10ffd8e64a33bbd950a140c7c514
I20240502 20:46:06.259109 7297 ts_tablet_manager.cc:1909] T 371e10ffd8e64a33bbd950a140c7c514 P c92e6e46782849479c8a0c6d82276fb2: Deleting consensus metadata
I20240502 20:46:06.259370 4618 tablet_service.cc:1514] Processing DeleteTablet for tablet 85e3a77b9daf4f3f93a10b223b168064 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:06 UTC) from {username='slave'} at 127.0.0.1:57068
I20240502 20:46:06.259393 4222 catalog_manager.cc:4830] TS ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193): tablet 371e10ffd8e64a33bbd950a140c7c514 (table rebalance_test_table_2 [id=59de2d64647446ab856b853639a3b1ca]) successfully deleted
I20240502 20:46:06.259562 4750 tablet_service.cc:1514] Processing DeleteTablet for tablet bdac13ec12c24d28abcca7f1e451451d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:06 UTC) from {username='slave'} at 127.0.0.1:38526
I20240502 20:46:06.259649 7297 tablet_replica.cc:330] T 85e3a77b9daf4f3f93a10b223b168064 P c92e6e46782849479c8a0c6d82276fb2: stopping tablet replica
I20240502 20:46:06.259737 7296 tablet_replica.cc:330] T 85e3a77b9daf4f3f93a10b223b168064 P 923cfab926e7443f8ffd7b21ad0a9d91: stopping tablet replica
I20240502 20:46:06.259811 7299 tablet_replica.cc:330] T bdac13ec12c24d28abcca7f1e451451d P ed47c023452a497192f35d2f617bf9bd: stopping tablet replica
I20240502 20:46:06.259881 7296 raft_consensus.cc:2229] T 85e3a77b9daf4f3f93a10b223b168064 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:06.259922 7299 raft_consensus.cc:2229] T bdac13ec12c24d28abcca7f1e451451d P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:06.260010 7296 raft_consensus.cc:2258] T 85e3a77b9daf4f3f93a10b223b168064 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:06.260097 4488 tablet_service.cc:1514] Processing DeleteTablet for tablet 371e10ffd8e64a33bbd950a140c7c514 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:06 UTC) from {username='slave'} at 127.0.0.1:48890
I20240502 20:46:06.260165 4222 catalog_manager.cc:4830] TS c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827): tablet 371e10ffd8e64a33bbd950a140c7c514 (table rebalance_test_table_2 [id=59de2d64647446ab856b853639a3b1ca]) successfully deleted
I20240502 20:46:06.260295 4357 tablet_service.cc:1514] Processing DeleteTablet for tablet bdac13ec12c24d28abcca7f1e451451d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:06 UTC) from {username='slave'} at 127.0.0.1:38306
I20240502 20:46:06.260462 7295 tablet_replica.cc:330] T bdac13ec12c24d28abcca7f1e451451d P c3d26eafd51547acb780d4aac5ebe7cc: stopping tablet replica
I20240502 20:46:06.260501 7296 ts_tablet_manager.cc:1875] T 85e3a77b9daf4f3f93a10b223b168064 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:06.260600 7295 raft_consensus.cc:2229] T bdac13ec12c24d28abcca7f1e451451d P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:06.260893 7295 raft_consensus.cc:2258] T bdac13ec12c24d28abcca7f1e451451d P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:06.261426 7295 ts_tablet_manager.cc:1875] T bdac13ec12c24d28abcca7f1e451451d P c3d26eafd51547acb780d4aac5ebe7cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:06.261478 4749 tablet_service.cc:1514] Processing DeleteTablet for tablet 85e3a77b9daf4f3f93a10b223b168064 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:06 UTC) from {username='slave'} at 127.0.0.1:38526
I20240502 20:46:06.262454 7296 ts_tablet_manager.cc:1888] T 85e3a77b9daf4f3f93a10b223b168064 P 923cfab926e7443f8ffd7b21ad0a9d91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 20:46:06.262528 7296 log.cc:1196] T 85e3a77b9daf4f3f93a10b223b168064 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal/wals/85e3a77b9daf4f3f93a10b223b168064
I20240502 20:46:06.262521 4230 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: rebalancer_dynamic_table [id=c3e6475692944a73a98468ce42b4c70a] alter complete (version 1)
I20240502 20:46:06.263731 7299 raft_consensus.cc:2258] T bdac13ec12c24d28abcca7f1e451451d P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:06.263787 7295 ts_tablet_manager.cc:1888] T bdac13ec12c24d28abcca7f1e451451d P c3d26eafd51547acb780d4aac5ebe7cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 20:46:06.263850 7295 log.cc:1196] T bdac13ec12c24d28abcca7f1e451451d P c3d26eafd51547acb780d4aac5ebe7cc: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal/wals/bdac13ec12c24d28abcca7f1e451451d
I20240502 20:46:06.264111 7295 ts_tablet_manager.cc:1909] T bdac13ec12c24d28abcca7f1e451451d P c3d26eafd51547acb780d4aac5ebe7cc: Deleting consensus metadata
I20240502 20:46:06.264061 7297 raft_consensus.cc:2229] T 85e3a77b9daf4f3f93a10b223b168064 P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:06.264472 7297 raft_consensus.cc:2258] T 85e3a77b9daf4f3f93a10b223b168064 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:06.264735 7295 tablet_replica.cc:330] T 371e10ffd8e64a33bbd950a140c7c514 P c3d26eafd51547acb780d4aac5ebe7cc: stopping tablet replica
I20240502 20:46:06.264858 7295 raft_consensus.cc:2229] T 371e10ffd8e64a33bbd950a140c7c514 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:06.264940 7297 ts_tablet_manager.cc:1875] T 85e3a77b9daf4f3f93a10b223b168064 P c92e6e46782849479c8a0c6d82276fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:06.264972 7295 raft_consensus.cc:2258] T 371e10ffd8e64a33bbd950a140c7c514 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:06.265231 4221 catalog_manager.cc:4830] TS c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097): tablet bdac13ec12c24d28abcca7f1e451451d (table rebalance_test_table_2 [id=59de2d64647446ab856b853639a3b1ca]) successfully deleted
I20240502 20:46:06.265498 7299 ts_tablet_manager.cc:1875] T bdac13ec12c24d28abcca7f1e451451d P ed47c023452a497192f35d2f617bf9bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:06.265542 7296 ts_tablet_manager.cc:1909] T 85e3a77b9daf4f3f93a10b223b168064 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting consensus metadata
I20240502 20:46:06.266142 7398 mvcc.cc:205] Tried to move back new op lower bound from 7023340610526703616 to 7023340610044256256. Current Snapshot: MvccSnapshot[applied={T|T < 7023340610526703616}]
I20240502 20:46:06.266290 4219 catalog_manager.cc:4830] TS 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141): tablet 85e3a77b9daf4f3f93a10b223b168064 (table rebalance_test_table_2 [id=59de2d64647446ab856b853639a3b1ca]) successfully deleted
I20240502 20:46:06.266806 4229 catalog_manager.cc:5576] TS a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841) does not have the latest schema for tablet 2a9efc56d59c444ea3d3338499e7a91e (table rebalancer_dynamic_table [id=c3e6475692944a73a98468ce42b4c70a]). Expected version 1 got 0
I20240502 20:46:06.266841 7295 ts_tablet_manager.cc:1875] T 371e10ffd8e64a33bbd950a140c7c514 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:06.267787 7299 ts_tablet_manager.cc:1888] T bdac13ec12c24d28abcca7f1e451451d P ed47c023452a497192f35d2f617bf9bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 20:46:06.267860 7299 log.cc:1196] T bdac13ec12c24d28abcca7f1e451451d P ed47c023452a497192f35d2f617bf9bd: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal/wals/bdac13ec12c24d28abcca7f1e451451d
I20240502 20:46:06.268121 7299 ts_tablet_manager.cc:1909] T bdac13ec12c24d28abcca7f1e451451d P ed47c023452a497192f35d2f617bf9bd: Deleting consensus metadata
I20240502 20:46:06.268839 4222 catalog_manager.cc:4830] TS ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193): tablet bdac13ec12c24d28abcca7f1e451451d (table rebalance_test_table_2 [id=59de2d64647446ab856b853639a3b1ca]) successfully deleted
I20240502 20:46:06.268733 7402 tablet.cc:1722] T 2a9efc56d59c444ea3d3338499e7a91e P a47e5c56f13843018ac584914a8d4707: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 20:46:06.269656 7295 ts_tablet_manager.cc:1888] T 371e10ffd8e64a33bbd950a140c7c514 P c3d26eafd51547acb780d4aac5ebe7cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 20:46:06.269783 7295 log.cc:1196] T 371e10ffd8e64a33bbd950a140c7c514 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal/wals/371e10ffd8e64a33bbd950a140c7c514
I20240502 20:46:06.270052 7295 ts_tablet_manager.cc:1909] T 371e10ffd8e64a33bbd950a140c7c514 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting consensus metadata
I20240502 20:46:06.270087 7402 tablet.cc:1620] T 2a9efc56d59c444ea3d3338499e7a91e P a47e5c56f13843018ac584914a8d4707: MemRowSet was empty: no flush needed.
I20240502 20:46:06.270727 4221 catalog_manager.cc:4830] TS c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097): tablet 371e10ffd8e64a33bbd950a140c7c514 (table rebalance_test_table_2 [id=59de2d64647446ab856b853639a3b1ca]) successfully deleted
I20240502 20:46:06.271859 7297 ts_tablet_manager.cc:1888] T 85e3a77b9daf4f3f93a10b223b168064 P c92e6e46782849479c8a0c6d82276fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 20:46:06.271945 7297 log.cc:1196] T 85e3a77b9daf4f3f93a10b223b168064 P c92e6e46782849479c8a0c6d82276fb2: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal/wals/85e3a77b9daf4f3f93a10b223b168064
I20240502 20:46:06.272203 7297 ts_tablet_manager.cc:1909] T 85e3a77b9daf4f3f93a10b223b168064 P c92e6e46782849479c8a0c6d82276fb2: Deleting consensus metadata
I20240502 20:46:06.272787 7297 tablet_replica.cc:330] T bdac13ec12c24d28abcca7f1e451451d P c92e6e46782849479c8a0c6d82276fb2: stopping tablet replica
I20240502 20:46:06.272944 7297 raft_consensus.cc:2229] T bdac13ec12c24d28abcca7f1e451451d P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:06.272933 4222 catalog_manager.cc:4830] TS c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827): tablet 85e3a77b9daf4f3f93a10b223b168064 (table rebalance_test_table_2 [id=59de2d64647446ab856b853639a3b1ca]) successfully deleted
I20240502 20:46:06.273094 7297 raft_consensus.cc:2258] T bdac13ec12c24d28abcca7f1e451451d P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:06.273380 7299 tablet_replica.cc:330] T 85e3a77b9daf4f3f93a10b223b168064 P ed47c023452a497192f35d2f617bf9bd: stopping tablet replica
I20240502 20:46:06.273509 7299 raft_consensus.cc:2229] T 85e3a77b9daf4f3f93a10b223b168064 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:06.273614 7299 raft_consensus.cc:2258] T 85e3a77b9daf4f3f93a10b223b168064 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:06.273633 4508 consensus_queue.cc:237] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } }
I20240502 20:46:06.274107 7299 ts_tablet_manager.cc:1875] T 85e3a77b9daf4f3f93a10b223b168064 P ed47c023452a497192f35d2f617bf9bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:06.275224 4374 raft_consensus.cc:1261] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:06.275660 4768 raft_consensus.cc:1261] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:06.276074 7299 ts_tablet_manager.cc:1888] T 85e3a77b9daf4f3f93a10b223b168064 P ed47c023452a497192f35d2f617bf9bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 20:46:06.276108 6414 consensus_queue.cc:1035] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:06.276293 7404 consensus_queue.cc:1035] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:06.276510 7297 ts_tablet_manager.cc:1875] T bdac13ec12c24d28abcca7f1e451451d P c92e6e46782849479c8a0c6d82276fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:06.277267 7299 log.cc:1196] T 85e3a77b9daf4f3f93a10b223b168064 P ed47c023452a497192f35d2f617bf9bd: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal/wals/85e3a77b9daf4f3f93a10b223b168064
I20240502 20:46:06.277652 7299 ts_tablet_manager.cc:1909] T 85e3a77b9daf4f3f93a10b223b168064 P ed47c023452a497192f35d2f617bf9bd: Deleting consensus metadata
I20240502 20:46:06.278316 4222 catalog_manager.cc:4830] TS ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193): tablet 85e3a77b9daf4f3f93a10b223b168064 (table rebalance_test_table_2 [id=59de2d64647446ab856b853639a3b1ca]) successfully deleted
I20240502 20:46:06.278733 7297 ts_tablet_manager.cc:1888] T bdac13ec12c24d28abcca7f1e451451d P c92e6e46782849479c8a0c6d82276fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 20:46:06.278798 7297 log.cc:1196] T bdac13ec12c24d28abcca7f1e451451d P c92e6e46782849479c8a0c6d82276fb2: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal/wals/bdac13ec12c24d28abcca7f1e451451d
I20240502 20:46:06.279006 7297 ts_tablet_manager.cc:1909] T bdac13ec12c24d28abcca7f1e451451d P c92e6e46782849479c8a0c6d82276fb2: Deleting consensus metadata
I20240502 20:46:06.279719 4222 catalog_manager.cc:4830] TS c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827): tablet bdac13ec12c24d28abcca7f1e451451d (table rebalance_test_table_2 [id=59de2d64647446ab856b853639a3b1ca]) successfully deleted
I20240502 20:46:06.281514 6414 raft_consensus.cc:2932] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } } }
W20240502 20:46:06.281877 4442 consensus_peers.cc:485] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867): Couldn't send request to peer a4bf1df7fa6b4948be502c182bfbb833. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 61a6d9a8e21e4a098cfad7281f0e0fd6. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:06.282299 4768 raft_consensus.cc:2932] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } } }
I20240502 20:46:06.283643 4374 raft_consensus.cc:2932] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } } }
I20240502 20:46:06.284653 4229 catalog_manager.cc:5484] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: config changed from index -1 to 4, NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) added. New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:06.295421 4768 consensus_queue.cc:237] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } }
I20240502 20:46:06.297241 4638 raft_consensus.cc:1261] T a2e81d2c63cd4e4cac5086506b5c4a12 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:06.297430 4508 raft_consensus.cc:1261] T a2e81d2c63cd4e4cac5086506b5c4a12 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:06.297770 7318 consensus_queue.cc:1035] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:06.297971 7300 consensus_queue.cc:1035] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:06.300364 7318 raft_consensus.cc:2932] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } } }
I20240502 20:46:06.301963 4638 raft_consensus.cc:2932] T a2e81d2c63cd4e4cac5086506b5c4a12 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } } }
I20240502 20:46:06.303561 4508 raft_consensus.cc:2932] T a2e81d2c63cd4e4cac5086506b5c4a12 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } } }
W20240502 20:46:06.304287 4705 consensus_peers.cc:485] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a2e81d2c63cd4e4cac5086506b5c4a12. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:06.305424 4230 catalog_manager.cc:5484] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd reported cstate change: config changed from index -1 to 4, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:06.311933 4768 consensus_queue.cc:237] T f352efa74d3c4da3907efdc37237211f P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } }
I20240502 20:46:06.313746 4638 raft_consensus.cc:1261] T f352efa74d3c4da3907efdc37237211f P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:06.314162 7318 consensus_queue.cc:1035] T f352efa74d3c4da3907efdc37237211f P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:06.314410 4508 raft_consensus.cc:1261] T f352efa74d3c4da3907efdc37237211f P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:06.314780 7318 consensus_queue.cc:1035] T f352efa74d3c4da3907efdc37237211f P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:06.316097 7300 raft_consensus.cc:2932] T f352efa74d3c4da3907efdc37237211f P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } } }
I20240502 20:46:06.316521 4638 raft_consensus.cc:2932] T f352efa74d3c4da3907efdc37237211f P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } } }
W20240502 20:46:06.319285 4703 consensus_peers.cc:485] T f352efa74d3c4da3907efdc37237211f P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f352efa74d3c4da3907efdc37237211f. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:06.319655 4508 raft_consensus.cc:2932] T f352efa74d3c4da3907efdc37237211f P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } } }
I20240502 20:46:06.320166 4229 catalog_manager.cc:5484] T f352efa74d3c4da3907efdc37237211f P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index -1 to 4, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } } }
I20240502 20:46:06.327777 4374 consensus_queue.cc:237] T f6d4a3019ba54eb8b3506b88f04a6262 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } }
I20240502 20:46:06.329447 4508 raft_consensus.cc:1261] T f6d4a3019ba54eb8b3506b88f04a6262 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:06.329566 4768 raft_consensus.cc:1261] T f6d4a3019ba54eb8b3506b88f04a6262 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:06.329921 7433 consensus_queue.cc:1035] T f6d4a3019ba54eb8b3506b88f04a6262 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:06.330137 7432 consensus_queue.cc:1035] T f6d4a3019ba54eb8b3506b88f04a6262 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:06.331902 7433 raft_consensus.cc:2932] T f6d4a3019ba54eb8b3506b88f04a6262 P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } } }
I20240502 20:46:06.332453 4508 raft_consensus.cc:2932] T f6d4a3019ba54eb8b3506b88f04a6262 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } } }
I20240502 20:46:06.332494 4768 raft_consensus.cc:2932] T f6d4a3019ba54eb8b3506b88f04a6262 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } } }
I20240502 20:46:06.335625 4235 catalog_manager.cc:5484] T f6d4a3019ba54eb8b3506b88f04a6262 P c92e6e46782849479c8a0c6d82276fb2 reported cstate change: config changed from index -1 to 4, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New cstate: current_term: 1 leader_uuid: "c92e6e46782849479c8a0c6d82276fb2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 20:46:06.337373 4313 consensus_peers.cc:485] T f6d4a3019ba54eb8b3506b88f04a6262 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f6d4a3019ba54eb8b3506b88f04a6262. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:06.343428 4508 consensus_queue.cc:237] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } }
I20240502 20:46:06.344894 4374 raft_consensus.cc:1261] T d37730b109d04b6395f75c7552524877 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:06.344895 4638 raft_consensus.cc:1261] T d37730b109d04b6395f75c7552524877 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:06.345027 6613 consensus_queue.cc:1035] T d6c2626b3ee7493c8a349f08e17082b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:06.345345 6414 consensus_queue.cc:1035] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:06.345556 7408 consensus_queue.cc:1035] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 20:46:06.345682 4442 consensus_peers.cc:485] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867): Couldn't send request to peer a4bf1df7fa6b4948be502c182bfbb833. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d37730b109d04b6395f75c7552524877. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:06.347707 7404 raft_consensus.cc:2932] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } } }
I20240502 20:46:06.348253 4374 raft_consensus.cc:2932] T d37730b109d04b6395f75c7552524877 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } } }
I20240502 20:46:06.350925 7017 consensus_queue.cc:1035] T d6c2626b3ee7493c8a349f08e17082b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:06.354768 4230 catalog_manager.cc:5484] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: config changed from index -1 to 4, NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) added. New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:06.355046 4638 raft_consensus.cc:2932] T d37730b109d04b6395f75c7552524877 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } } }
I20240502 20:46:06.361828 7310 consensus_queue.cc:1035] T 8bbf33f0ec4a4adb9a979b08606ee822 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:06.362447 4508 consensus_queue.cc:237] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } }
I20240502 20:46:06.364089 4374 raft_consensus.cc:1261] T a62a7ce526f14c25994bb9e0e3c3bc84 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:06.364527 7408 consensus_queue.cc:1035] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:06.364929 4637 raft_consensus.cc:1261] T a62a7ce526f14c25994bb9e0e3c3bc84 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:06.365911 7408 consensus_queue.cc:1035] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:06.366386 4235 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 20:46:06.367606 7310 consensus_queue.cc:1035] T 8bbf33f0ec4a4adb9a979b08606ee822 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:06.367779 7408 raft_consensus.cc:2932] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } } }
I20240502 20:46:06.370106 4637 raft_consensus.cc:2932] T a62a7ce526f14c25994bb9e0e3c3bc84 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } } }
W20240502 20:46:06.371260 4443 consensus_peers.cc:485] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a62a7ce526f14c25994bb9e0e3c3bc84. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:06.373725 4235 catalog_manager.cc:5484] T a62a7ce526f14c25994bb9e0e3c3bc84 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index -1 to 4, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } } }
I20240502 20:46:06.373807 4374 raft_consensus.cc:2932] T a62a7ce526f14c25994bb9e0e3c3bc84 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } } }
I20240502 20:46:06.374205 7234 tablet.cc:1722] T 2a9efc56d59c444ea3d3338499e7a91e P a4bf1df7fa6b4948be502c182bfbb833: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 20:46:06.374550 7234 tablet.cc:1620] T 2a9efc56d59c444ea3d3338499e7a91e P a4bf1df7fa6b4948be502c182bfbb833: MemRowSet was empty: no flush needed.
I20240502 20:46:06.374590 7242 tablet.cc:1722] T 2a9efc56d59c444ea3d3338499e7a91e P 7845c397230d49358ba2fad9db53f97c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 20:46:06.374891 7242 tablet.cc:1620] T 2a9efc56d59c444ea3d3338499e7a91e P 7845c397230d49358ba2fad9db53f97c: MemRowSet was empty: no flush needed.
I20240502 20:46:06.381197 7402 tablet.cc:1722] T 2a9efc56d59c444ea3d3338499e7a91e P a47e5c56f13843018ac584914a8d4707: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 20:46:06.381513 7402 tablet.cc:1620] T 2a9efc56d59c444ea3d3338499e7a91e P a47e5c56f13843018ac584914a8d4707: MemRowSet was empty: no flush needed.
I20240502 20:46:06.381825 4230 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: rebalancer_dynamic_table [id=c3e6475692944a73a98468ce42b4c70a] alter complete (version 2)
I20240502 20:46:06.398928 4508 consensus_queue.cc:237] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } }
I20240502 20:46:06.400792 4637 raft_consensus.cc:1261] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:06.401216 4377 raft_consensus.cc:1261] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:06.403008 6414 consensus_queue.cc:1035] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:06.403594 6414 consensus_queue.cc:1035] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 20:46:06.404017 4441 consensus_peers.cc:485] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5a36bf6e8e6e43b2b47b6d6357b0d445. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:06.404909 7404 raft_consensus.cc:2932] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } } }
I20240502 20:46:06.405826 4637 raft_consensus.cc:2932] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } } }
I20240502 20:46:06.407953 4377 raft_consensus.cc:2932] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } } }
I20240502 20:46:06.410265 4230 catalog_manager.cc:5484] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: config changed from index -1 to 4, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:06.420451 4637 consensus_queue.cc:237] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } }
I20240502 20:46:06.422022 4768 raft_consensus.cc:1261] T 8565ebbd21dd4f708a0254d1ed2d6670 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:06.422145 4508 raft_consensus.cc:1261] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 20:46:06.422168 4575 consensus_peers.cc:485] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8565ebbd21dd4f708a0254d1ed2d6670. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:06.422524 7017 consensus_queue.cc:1035] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:06.422757 7447 consensus_queue.cc:1035] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:06.424546 6613 raft_consensus.cc:2932] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } } }
I20240502 20:46:06.425065 4508 raft_consensus.cc:2932] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } } }
I20240502 20:46:06.425061 4768 raft_consensus.cc:2932] T 8565ebbd21dd4f708a0254d1ed2d6670 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } } }
I20240502 20:46:06.428079 4233 catalog_manager.cc:5484] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index -1 to 4, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:06.435263 4768 consensus_queue.cc:237] T 02bddc1866e04858894e047ec100a6f0 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } }
I20240502 20:46:06.436722 4637 raft_consensus.cc:1261] T 02bddc1866e04858894e047ec100a6f0 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:06.436738 4377 raft_consensus.cc:1261] T 02bddc1866e04858894e047ec100a6f0 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:06.437306 7318 consensus_queue.cc:1035] T 02bddc1866e04858894e047ec100a6f0 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:06.437476 6241 consensus_queue.cc:1035] T 02bddc1866e04858894e047ec100a6f0 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:06.439409 6241 raft_consensus.cc:2932] T 02bddc1866e04858894e047ec100a6f0 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } } }
I20240502 20:46:06.441051 4377 raft_consensus.cc:2932] T 02bddc1866e04858894e047ec100a6f0 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } } }
W20240502 20:46:06.441231 4704 consensus_peers.cc:485] T 02bddc1866e04858894e047ec100a6f0 P ed47c023452a497192f35d2f617bf9bd -> Peer a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867): Couldn't send request to peer a4bf1df7fa6b4948be502c182bfbb833. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 02bddc1866e04858894e047ec100a6f0. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:06.441445 4637 raft_consensus.cc:2932] T 02bddc1866e04858894e047ec100a6f0 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } } }
I20240502 20:46:06.443902 4235 catalog_manager.cc:5484] T 02bddc1866e04858894e047ec100a6f0 P ed47c023452a497192f35d2f617bf9bd reported cstate change: config changed from index -1 to 4, NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) added. New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 20:46:06.452108 6414 consensus_queue.cc:1159] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" responder_term: 1 status { last_received { term: 1 index: 3 } last_committed_idx: 3 last_received_current_leader { term: 1 index: 3 } }
I20240502 20:46:06.452345 4508 consensus_queue.cc:237] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } }
I20240502 20:46:06.453872 4637 raft_consensus.cc:1261] T d6fb394fdcff436886fe0634d79c024c P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 20:46:06.454054 4443 consensus_peers.cc:485] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d6fb394fdcff436886fe0634d79c024c. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:06.454084 4768 raft_consensus.cc:1261] T d6fb394fdcff436886fe0634d79c024c P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:06.454306 6414 consensus_queue.cc:1035] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:06.454479 6375 consensus_queue.cc:1035] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:06.456264 7404 raft_consensus.cc:2932] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } } }
I20240502 20:46:06.456723 4637 raft_consensus.cc:2932] T d6fb394fdcff436886fe0634d79c024c P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } } }
I20240502 20:46:06.456787 4768 raft_consensus.cc:2932] T d6fb394fdcff436886fe0634d79c024c P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } } }
I20240502 20:46:06.459673 4230 catalog_manager.cc:5484] T d6fb394fdcff436886fe0634d79c024c P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index -1 to 4, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } } }
I20240502 20:46:06.466957 4768 consensus_queue.cc:237] T 9e34e8e17a7e413b97eb614a7df727d5 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } }
I20240502 20:46:06.468315 4508 raft_consensus.cc:1261] T 9e34e8e17a7e413b97eb614a7df727d5 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:06.468315 4377 raft_consensus.cc:1261] T 9e34e8e17a7e413b97eb614a7df727d5 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 20:46:06.468493 4703 consensus_peers.cc:485] T 9e34e8e17a7e413b97eb614a7df727d5 P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9e34e8e17a7e413b97eb614a7df727d5. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:06.468765 7318 consensus_queue.cc:1035] T 9e34e8e17a7e413b97eb614a7df727d5 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:06.468959 6241 consensus_queue.cc:1035] T 9e34e8e17a7e413b97eb614a7df727d5 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:06.471133 7318 raft_consensus.cc:2932] T 9e34e8e17a7e413b97eb614a7df727d5 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } } }
I20240502 20:46:06.471871 4506 raft_consensus.cc:2932] T 9e34e8e17a7e413b97eb614a7df727d5 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } } }
I20240502 20:46:06.471905 4374 raft_consensus.cc:2932] T 9e34e8e17a7e413b97eb614a7df727d5 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } } }
I20240502 20:46:06.475116 4228 catalog_manager.cc:5484] T 9e34e8e17a7e413b97eb614a7df727d5 P ed47c023452a497192f35d2f617bf9bd reported cstate change: config changed from index -1 to 4, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:06.482821 4374 consensus_queue.cc:237] T 47c5087093214fbeae35590b4b3c3d42 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } }
I20240502 20:46:06.484391 4768 raft_consensus.cc:1261] T 47c5087093214fbeae35590b4b3c3d42 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:06.484392 4506 raft_consensus.cc:1261] T 47c5087093214fbeae35590b4b3c3d42 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:06.484818 6138 consensus_queue.cc:1035] T 47c5087093214fbeae35590b4b3c3d42 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 20:46:06.485018 4313 consensus_peers.cc:485] T 47c5087093214fbeae35590b4b3c3d42 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 47c5087093214fbeae35590b4b3c3d42. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:06.485198 6138 consensus_queue.cc:1035] T 47c5087093214fbeae35590b4b3c3d42 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:06.486624 4235 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 20:46:06.486969 7433 raft_consensus.cc:2932] T 47c5087093214fbeae35590b4b3c3d42 P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } } }
I20240502 20:46:06.487295 4768 raft_consensus.cc:2932] T 47c5087093214fbeae35590b4b3c3d42 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } } }
I20240502 20:46:06.490883 4506 raft_consensus.cc:2932] T 47c5087093214fbeae35590b4b3c3d42 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } } }
I20240502 20:46:06.490948 4230 catalog_manager.cc:5484] T 47c5087093214fbeae35590b4b3c3d42 P c92e6e46782849479c8a0c6d82276fb2 reported cstate change: config changed from index -1 to 4, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New cstate: current_term: 1 leader_uuid: "c92e6e46782849479c8a0c6d82276fb2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:06.495373 7234 tablet.cc:1722] T 2a9efc56d59c444ea3d3338499e7a91e P a4bf1df7fa6b4948be502c182bfbb833: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 20:46:06.495963 7402 tablet.cc:1722] T 2a9efc56d59c444ea3d3338499e7a91e P a47e5c56f13843018ac584914a8d4707: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 20:46:06.497668 7242 tablet.cc:1722] T 2a9efc56d59c444ea3d3338499e7a91e P 7845c397230d49358ba2fad9db53f97c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 20:46:06.501912 4637 consensus_queue.cc:237] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } }
I20240502 20:46:06.502542 4235 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: rebalancer_dynamic_table_new_name [id=c3e6475692944a73a98468ce42b4c70a] alter complete (version 3)
W20240502 20:46:06.503540 4573 consensus_peers.cc:485] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867): Couldn't send request to peer a4bf1df7fa6b4948be502c182bfbb833. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7abfd61f55554042978b50761cbd6ab5. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:06.503717 4768 raft_consensus.cc:1261] T 7abfd61f55554042978b50761cbd6ab5 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:06.504107 4374 raft_consensus.cc:1261] T 7abfd61f55554042978b50761cbd6ab5 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:06.504235 6613 consensus_queue.cc:1035] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:06.504480 7017 consensus_queue.cc:1035] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:06.506098 6613 raft_consensus.cc:2932] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } } }
I20240502 20:46:06.506510 4768 raft_consensus.cc:2932] T 7abfd61f55554042978b50761cbd6ab5 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } } }
I20240502 20:46:06.506732 4374 raft_consensus.cc:2932] T 7abfd61f55554042978b50761cbd6ab5 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } } }
I20240502 20:46:06.509578 4230 catalog_manager.cc:5484] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index -1 to 4, NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) added. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:06.517194 4374 consensus_queue.cc:237] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } }
I20240502 20:46:06.518976 4637 raft_consensus.cc:1261] T 8e7e145762bf458a8374bfc503025a17 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:06.519150 4506 raft_consensus.cc:1261] T 8e7e145762bf458a8374bfc503025a17 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:06.519407 7433 consensus_queue.cc:1035] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:06.519605 6138 consensus_queue.cc:1035] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:06.521953 7433 raft_consensus.cc:2932] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } } }
W20240502 20:46:06.522938 4310 consensus_peers.cc:485] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8e7e145762bf458a8374bfc503025a17. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:06.522800 4506 raft_consensus.cc:2932] T 8e7e145762bf458a8374bfc503025a17 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } } }
I20240502 20:46:06.524266 4637 raft_consensus.cc:2932] T 8e7e145762bf458a8374bfc503025a17 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } } }
I20240502 20:46:06.525717 4235 catalog_manager.cc:5484] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2 reported cstate change: config changed from index -1 to 4, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New cstate: current_term: 1 leader_uuid: "c92e6e46782849479c8a0c6d82276fb2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:06.535715 4768 consensus_queue.cc:237] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } }
I20240502 20:46:06.537168 4637 raft_consensus.cc:1261] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:06.537379 4506 raft_consensus.cc:1261] T d37fab152a8a44868d3160ec3057bb98 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:06.537868 6241 consensus_queue.cc:1035] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:06.538100 7318 consensus_queue.cc:1035] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 20:46:06.538362 4705 consensus_peers.cc:485] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d37fab152a8a44868d3160ec3057bb98. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:06.539930 7300 raft_consensus.cc:2932] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } } }
I20240502 20:46:06.540350 4637 raft_consensus.cc:2932] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } } }
I20240502 20:46:06.540472 4506 raft_consensus.cc:2932] T d37fab152a8a44868d3160ec3057bb98 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } } }
I20240502 20:46:06.544126 4230 catalog_manager.cc:5484] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index -1 to 4, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } } }
I20240502 20:46:06.551828 6702 raft_consensus.cc:3037] T bcfd626ea0a04fa7b03dd0ede6b35266 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:06.552259 4506 consensus_queue.cc:237] T f09dcfea3eb347e687f3c72f4ccc6929 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } }
I20240502 20:46:06.553671 7310 consensus_queue.cc:1035] T bcfd626ea0a04fa7b03dd0ede6b35266 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:06.554433 4768 raft_consensus.cc:1261] T f09dcfea3eb347e687f3c72f4ccc6929 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:06.554543 4374 raft_consensus.cc:1261] T f09dcfea3eb347e687f3c72f4ccc6929 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:06.554868 6414 consensus_queue.cc:1035] T f09dcfea3eb347e687f3c72f4ccc6929 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:06.555030 7404 consensus_queue.cc:1035] T f09dcfea3eb347e687f3c72f4ccc6929 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:06.557301 7404 raft_consensus.cc:2932] T f09dcfea3eb347e687f3c72f4ccc6929 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } } }
I20240502 20:46:06.558755 4768 raft_consensus.cc:2932] T f09dcfea3eb347e687f3c72f4ccc6929 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } } }
I20240502 20:46:06.559756 4374 raft_consensus.cc:2932] T f09dcfea3eb347e687f3c72f4ccc6929 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } } }
W20240502 20:46:06.560501 4444 consensus_peers.cc:485] T f09dcfea3eb347e687f3c72f4ccc6929 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f09dcfea3eb347e687f3c72f4ccc6929. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:06.560842 4228 catalog_manager.cc:5484] T f09dcfea3eb347e687f3c72f4ccc6929 P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: config changed from index -1 to 4, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:06.561627 7322 consensus_queue.cc:1035] T bcfd626ea0a04fa7b03dd0ede6b35266 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:06.571849 4506 consensus_queue.cc:237] T 28029d4e66d1453cbde9811ffc2a3283 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } }
I20240502 20:46:06.573124 4374 raft_consensus.cc:1261] T 28029d4e66d1453cbde9811ffc2a3283 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 20:46:06.573347 4441 consensus_peers.cc:485] T 28029d4e66d1453cbde9811ffc2a3283 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 28029d4e66d1453cbde9811ffc2a3283. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:06.573616 6414 consensus_queue.cc:1035] T 28029d4e66d1453cbde9811ffc2a3283 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:06.574280 4637 raft_consensus.cc:1261] T 28029d4e66d1453cbde9811ffc2a3283 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:06.575330 6414 consensus_queue.cc:1035] T 28029d4e66d1453cbde9811ffc2a3283 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:06.576000 7408 raft_consensus.cc:2932] T 28029d4e66d1453cbde9811ffc2a3283 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } } }
I20240502 20:46:06.577247 4377 raft_consensus.cc:2932] T 28029d4e66d1453cbde9811ffc2a3283 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } } }
I20240502 20:46:06.577927 4637 raft_consensus.cc:2932] T 28029d4e66d1453cbde9811ffc2a3283 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } } }
I20240502 20:46:06.580376 4230 catalog_manager.cc:5484] T 28029d4e66d1453cbde9811ffc2a3283 P c92e6e46782849479c8a0c6d82276fb2 reported cstate change: config changed from index -1 to 4, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } } }
I20240502 20:46:06.591116 4506 consensus_queue.cc:237] T 61f8c8d2925d46bdad8e903b48c3046e P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } }
I20240502 20:46:06.592439 4637 raft_consensus.cc:1261] T 61f8c8d2925d46bdad8e903b48c3046e P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:06.592478 4377 raft_consensus.cc:1261] T 61f8c8d2925d46bdad8e903b48c3046e P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 20:46:06.592806 4442 consensus_peers.cc:485] T 61f8c8d2925d46bdad8e903b48c3046e P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867): Couldn't send request to peer a4bf1df7fa6b4948be502c182bfbb833. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 61f8c8d2925d46bdad8e903b48c3046e. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:06.592902 6414 consensus_queue.cc:1035] T 61f8c8d2925d46bdad8e903b48c3046e P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:06.593096 6375 consensus_queue.cc:1035] T 61f8c8d2925d46bdad8e903b48c3046e P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:06.594724 6414 raft_consensus.cc:2932] T 61f8c8d2925d46bdad8e903b48c3046e P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } } }
I20240502 20:46:06.595254 4377 raft_consensus.cc:2932] T 61f8c8d2925d46bdad8e903b48c3046e P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } } }
I20240502 20:46:06.595316 4637 raft_consensus.cc:2932] T 61f8c8d2925d46bdad8e903b48c3046e P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } } }
I20240502 20:46:06.598577 4230 catalog_manager.cc:5484] T 61f8c8d2925d46bdad8e903b48c3046e P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index -1 to 4, NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) added. New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } } }
I20240502 20:46:06.605695 4637 consensus_queue.cc:237] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } }
I20240502 20:46:06.606482 4233 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 20:46:06.606611 4233 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 20:46:06.607250 4768 raft_consensus.cc:1261] T 00eb91e624554baf918e96b098019be8 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:06.607540 4506 raft_consensus.cc:1261] T 00eb91e624554baf918e96b098019be8 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:06.607707 7017 consensus_queue.cc:1035] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:06.607916 7447 consensus_queue.cc:1035] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:06.609068 4233 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: Sending DeleteTablet for 3 replicas of tablet 2a9efc56d59c444ea3d3338499e7a91e
I20240502 20:46:06.609705 6946 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a9efc56d59c444ea3d3338499e7a91e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:06 UTC) from {username='slave'} at 127.0.0.1:45156
I20240502 20:46:06.609985 6809 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a9efc56d59c444ea3d3338499e7a91e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:06 UTC) from {username='slave'} at 127.0.0.1:35290
I20240502 20:46:06.610389 7570 tablet_replica.cc:330] T 2a9efc56d59c444ea3d3338499e7a91e P a4bf1df7fa6b4948be502c182bfbb833: stopping tablet replica
I20240502 20:46:06.610525 7570 raft_consensus.cc:2229] T 2a9efc56d59c444ea3d3338499e7a91e P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:06.610769 7570 raft_consensus.cc:2258] T 2a9efc56d59c444ea3d3338499e7a91e P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:06.610770 6683 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a9efc56d59c444ea3d3338499e7a91e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:06 UTC) from {username='slave'} at 127.0.0.1:37506
I20240502 20:46:06.610950 7017 raft_consensus.cc:2932] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } } }
I20240502 20:46:06.611119 7572 tablet_replica.cc:330] T 2a9efc56d59c444ea3d3338499e7a91e P 7845c397230d49358ba2fad9db53f97c: stopping tablet replica
I20240502 20:46:06.611188 7569 tablet_replica.cc:330] T 2a9efc56d59c444ea3d3338499e7a91e P a47e5c56f13843018ac584914a8d4707: stopping tablet replica
I20240502 20:46:06.611356 7572 raft_consensus.cc:2229] T 2a9efc56d59c444ea3d3338499e7a91e P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:06.611505 7570 ts_tablet_manager.cc:1875] T 2a9efc56d59c444ea3d3338499e7a91e P a4bf1df7fa6b4948be502c182bfbb833: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:06.611505 7569 raft_consensus.cc:2229] T 2a9efc56d59c444ea3d3338499e7a91e P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:06.611522 7572 raft_consensus.cc:2258] T 2a9efc56d59c444ea3d3338499e7a91e P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:06.611665 7569 raft_consensus.cc:2258] T 2a9efc56d59c444ea3d3338499e7a91e P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:06.612962 7569 ts_tablet_manager.cc:1875] T 2a9efc56d59c444ea3d3338499e7a91e P a47e5c56f13843018ac584914a8d4707: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:06.612967 4506 raft_consensus.cc:2932] T 00eb91e624554baf918e96b098019be8 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } } }
I20240502 20:46:06.613667 7570 ts_tablet_manager.cc:1888] T 2a9efc56d59c444ea3d3338499e7a91e P a4bf1df7fa6b4948be502c182bfbb833: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:06.613739 7570 log.cc:1196] T 2a9efc56d59c444ea3d3338499e7a91e P a4bf1df7fa6b4948be502c182bfbb833: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/wal/wals/2a9efc56d59c444ea3d3338499e7a91e
I20240502 20:46:06.614076 7570 ts_tablet_manager.cc:1909] T 2a9efc56d59c444ea3d3338499e7a91e P a4bf1df7fa6b4948be502c182bfbb833: Deleting consensus metadata
I20240502 20:46:06.614537 7572 ts_tablet_manager.cc:1875] T 2a9efc56d59c444ea3d3338499e7a91e P 7845c397230d49358ba2fad9db53f97c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:06.614781 4220 catalog_manager.cc:4830] TS a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867): tablet 2a9efc56d59c444ea3d3338499e7a91e (table rebalancer_dynamic_table_new_name [id=c3e6475692944a73a98468ce42b4c70a]) successfully deleted
I20240502 20:46:06.615401 4233 catalog_manager.cc:5484] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index -1 to 4, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:06.617621 7572 ts_tablet_manager.cc:1888] T 2a9efc56d59c444ea3d3338499e7a91e P 7845c397230d49358ba2fad9db53f97c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:06.617758 7572 log.cc:1196] T 2a9efc56d59c444ea3d3338499e7a91e P 7845c397230d49358ba2fad9db53f97c: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/wal/wals/2a9efc56d59c444ea3d3338499e7a91e
I20240502 20:46:06.618168 7572 ts_tablet_manager.cc:1909] T 2a9efc56d59c444ea3d3338499e7a91e P 7845c397230d49358ba2fad9db53f97c: Deleting consensus metadata
W20240502 20:46:06.618232 4574 consensus_peers.cc:485] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 00eb91e624554baf918e96b098019be8. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:06.618541 4768 raft_consensus.cc:2932] T 00eb91e624554baf918e96b098019be8 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } } }
I20240502 20:46:06.619000 4221 catalog_manager.cc:4830] TS 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): tablet 2a9efc56d59c444ea3d3338499e7a91e (table rebalancer_dynamic_table_new_name [id=c3e6475692944a73a98468ce42b4c70a]) successfully deleted
I20240502 20:46:06.620585 7569 ts_tablet_manager.cc:1888] T 2a9efc56d59c444ea3d3338499e7a91e P a47e5c56f13843018ac584914a8d4707: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:06.620685 7569 log.cc:1196] T 2a9efc56d59c444ea3d3338499e7a91e P a47e5c56f13843018ac584914a8d4707: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/wal/wals/2a9efc56d59c444ea3d3338499e7a91e
I20240502 20:46:06.621217 7569 ts_tablet_manager.cc:1909] T 2a9efc56d59c444ea3d3338499e7a91e P a47e5c56f13843018ac584914a8d4707: Deleting consensus metadata
I20240502 20:46:06.622426 4222 catalog_manager.cc:4830] TS a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): tablet 2a9efc56d59c444ea3d3338499e7a91e (table rebalancer_dynamic_table_new_name [id=c3e6475692944a73a98468ce42b4c70a]) successfully deleted
I20240502 20:46:06.625738 4637 consensus_queue.cc:237] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } }
I20240502 20:46:06.627104 4768 raft_consensus.cc:1261] T 428ffbac75db466191cb3bef63b4eb79 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:06.627267 4377 raft_consensus.cc:1261] T 428ffbac75db466191cb3bef63b4eb79 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:06.627527 7017 consensus_queue.cc:1035] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 20:46:06.627753 4575 consensus_peers.cc:485] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 428ffbac75db466191cb3bef63b4eb79. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:06.628031 7017 consensus_queue.cc:1035] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:06.629374 7447 raft_consensus.cc:2932] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } } }
I20240502 20:46:06.630977 4768 raft_consensus.cc:2932] T 428ffbac75db466191cb3bef63b4eb79 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } } }
I20240502 20:46:06.631357 4377 raft_consensus.cc:2932] T 428ffbac75db466191cb3bef63b4eb79 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } } }
I20240502 20:46:06.632211 4233 catalog_manager.cc:5484] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index -1 to 4, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:06.633836 4235 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41812:
name: "rebalancer_extra_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:46:06.641467 4749 tablet_service.cc:1467] Processing CreateTablet for tablet 180902058815460189f75de4704d1e84 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=44d455c601714afb8333ce2027e68a4a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:06.641467 6809 tablet_service.cc:1467] Processing CreateTablet for tablet 180902058815460189f75de4704d1e84 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=44d455c601714afb8333ce2027e68a4a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:06.641774 4749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 180902058815460189f75de4704d1e84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:06.641793 6809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 180902058815460189f75de4704d1e84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:06.641705 7078 tablet_service.cc:1467] Processing CreateTablet for tablet 180902058815460189f75de4704d1e84 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=44d455c601714afb8333ce2027e68a4a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:06.642024 7078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 180902058815460189f75de4704d1e84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:06.642611 4487 tablet_service.cc:1467] Processing CreateTablet for tablet 935f0efc1fe146f4ac21fac4c0cc688c (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=44d455c601714afb8333ce2027e68a4a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:06.642880 4487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 935f0efc1fe146f4ac21fac4c0cc688c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:06.644184 4488 tablet_service.cc:1467] Processing CreateTablet for tablet 15a2cbe8285a4938be7f628f7f8b734d (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=44d455c601714afb8333ce2027e68a4a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:06.644434 4488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15a2cbe8285a4938be7f628f7f8b734d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:06.645128 7079 tablet_service.cc:1467] Processing CreateTablet for tablet 935f0efc1fe146f4ac21fac4c0cc688c (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=44d455c601714afb8333ce2027e68a4a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:06.645293 7589 tablet_bootstrap.cc:492] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:46:06.645380 7079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 935f0efc1fe146f4ac21fac4c0cc688c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:06.645851 7587 tablet_bootstrap.cc:492] T 180902058815460189f75de4704d1e84 P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:46:06.646243 7589 tablet_bootstrap.cc:654] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:06.647846 7588 tablet_bootstrap.cc:492] T 180902058815460189f75de4704d1e84 P 201d2b6485084e7dad2cbb722af94779: Bootstrap starting.
I20240502 20:46:06.647939 7587 tablet_bootstrap.cc:654] T 180902058815460189f75de4704d1e84 P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:06.648823 7588 tablet_bootstrap.cc:654] T 180902058815460189f75de4704d1e84 P 201d2b6485084e7dad2cbb722af94779: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:06.649470 4357 tablet_service.cc:1467] Processing CreateTablet for tablet 15a2cbe8285a4938be7f628f7f8b734d (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=44d455c601714afb8333ce2027e68a4a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:06.649765 4357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15a2cbe8285a4938be7f628f7f8b734d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:06.649995 6946 tablet_service.cc:1467] Processing CreateTablet for tablet 935f0efc1fe146f4ac21fac4c0cc688c (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=44d455c601714afb8333ce2027e68a4a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:06.650218 6946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 935f0efc1fe146f4ac21fac4c0cc688c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:06.651665 7592 tablet_bootstrap.cc:492] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:46:06.652683 7078 tablet_service.cc:1467] Processing CreateTablet for tablet 15a2cbe8285a4938be7f628f7f8b734d (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=44d455c601714afb8333ce2027e68a4a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:06.653060 7592 tablet_bootstrap.cc:654] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:06.653163 7078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15a2cbe8285a4938be7f628f7f8b734d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:06.655689 7589 tablet_bootstrap.cc:492] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:46:06.655853 7589 ts_tablet_manager.cc:1367] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240502 20:46:06.655961 7592 tablet_bootstrap.cc:492] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:46:06.656046 7592 ts_tablet_manager.cc:1367] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 20:46:06.656589 7593 tablet_bootstrap.cc:492] T 935f0efc1fe146f4ac21fac4c0cc688c P a47e5c56f13843018ac584914a8d4707: Bootstrap starting.
I20240502 20:46:06.656565 7592 raft_consensus.cc:348] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:06.656678 7592 raft_consensus.cc:374] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:06.656708 7592 raft_consensus.cc:729] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:46:06.656843 7592 consensus_queue.cc:260] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:06.657181 7592 ts_tablet_manager.cc:1398] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:06.657230 7589 raft_consensus.cc:348] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:06.657344 7589 raft_consensus.cc:374] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:06.657377 7589 raft_consensus.cc:729] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:46:06.657480 7589 consensus_queue.cc:260] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:06.658636 7593 tablet_bootstrap.cc:654] T 935f0efc1fe146f4ac21fac4c0cc688c P a47e5c56f13843018ac584914a8d4707: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:06.659045 7589 ts_tablet_manager.cc:1398] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 20:46:06.659298 7589 tablet_bootstrap.cc:492] T 15a2cbe8285a4938be7f628f7f8b734d P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:46:06.660450 7586 tablet_bootstrap.cc:492] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap starting.
I20240502 20:46:06.661476 7589 tablet_bootstrap.cc:654] T 15a2cbe8285a4938be7f628f7f8b734d P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:06.662106 7586 tablet_bootstrap.cc:654] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:06.662793 7593 tablet_bootstrap.cc:492] T 935f0efc1fe146f4ac21fac4c0cc688c P a47e5c56f13843018ac584914a8d4707: No bootstrap required, opened a new log
I20240502 20:46:06.663548 7593 ts_tablet_manager.cc:1367] T 935f0efc1fe146f4ac21fac4c0cc688c P a47e5c56f13843018ac584914a8d4707: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 20:46:06.664094 7593 raft_consensus.cc:348] T 935f0efc1fe146f4ac21fac4c0cc688c P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:06.664374 7588 tablet_bootstrap.cc:492] T 180902058815460189f75de4704d1e84 P 201d2b6485084e7dad2cbb722af94779: No bootstrap required, opened a new log
I20240502 20:46:06.664525 7588 ts_tablet_manager.cc:1367] T 180902058815460189f75de4704d1e84 P 201d2b6485084e7dad2cbb722af94779: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20240502 20:46:06.664727 7586 tablet_bootstrap.cc:492] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833: No bootstrap required, opened a new log
I20240502 20:46:06.664832 7586 ts_tablet_manager.cc:1367] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 20:46:06.665035 7593 raft_consensus.cc:374] T 935f0efc1fe146f4ac21fac4c0cc688c P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:06.665146 7593 raft_consensus.cc:729] T 935f0efc1fe146f4ac21fac4c0cc688c P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Initialized, Role: FOLLOWER
I20240502 20:46:06.665154 7588 raft_consensus.cc:348] T 180902058815460189f75de4704d1e84 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:06.665315 7588 raft_consensus.cc:374] T 180902058815460189f75de4704d1e84 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:06.665387 7588 raft_consensus.cc:729] T 180902058815460189f75de4704d1e84 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Initialized, Role: FOLLOWER
I20240502 20:46:06.665344 7593 consensus_queue.cc:260] T 935f0efc1fe146f4ac21fac4c0cc688c P a47e5c56f13843018ac584914a8d4707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:06.665552 7588 consensus_queue.cc:260] T 180902058815460189f75de4704d1e84 P 201d2b6485084e7dad2cbb722af94779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:06.665688 7593 ts_tablet_manager.cc:1398] T 935f0efc1fe146f4ac21fac4c0cc688c P a47e5c56f13843018ac584914a8d4707: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:06.666479 7588 ts_tablet_manager.cc:1398] T 180902058815460189f75de4704d1e84 P 201d2b6485084e7dad2cbb722af94779: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:06.666695 7588 tablet_bootstrap.cc:492] T 935f0efc1fe146f4ac21fac4c0cc688c P 201d2b6485084e7dad2cbb722af94779: Bootstrap starting.
I20240502 20:46:06.667765 7588 tablet_bootstrap.cc:654] T 935f0efc1fe146f4ac21fac4c0cc688c P 201d2b6485084e7dad2cbb722af94779: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:06.668758 7586 raft_consensus.cc:348] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:06.668912 7586 raft_consensus.cc:374] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:06.668956 7586 raft_consensus.cc:729] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Initialized, Role: FOLLOWER
I20240502 20:46:06.669144 7586 consensus_queue.cc:260] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:06.669447 7586 ts_tablet_manager.cc:1398] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 20:46:06.670928 7587 tablet_bootstrap.cc:492] T 180902058815460189f75de4704d1e84 P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:46:06.671032 7587 ts_tablet_manager.cc:1367] T 180902058815460189f75de4704d1e84 P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.000s
I20240502 20:46:06.671455 7587 raft_consensus.cc:348] T 180902058815460189f75de4704d1e84 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:06.671563 7587 raft_consensus.cc:374] T 180902058815460189f75de4704d1e84 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:06.671600 7587 raft_consensus.cc:729] T 180902058815460189f75de4704d1e84 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:46:06.671703 7587 consensus_queue.cc:260] T 180902058815460189f75de4704d1e84 P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:06.672040 7587 ts_tablet_manager.cc:1398] T 180902058815460189f75de4704d1e84 P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:06.673873 7588 tablet_bootstrap.cc:492] T 935f0efc1fe146f4ac21fac4c0cc688c P 201d2b6485084e7dad2cbb722af94779: No bootstrap required, opened a new log
I20240502 20:46:06.674017 7588 ts_tablet_manager.cc:1367] T 935f0efc1fe146f4ac21fac4c0cc688c P 201d2b6485084e7dad2cbb722af94779: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 20:46:06.674610 7588 raft_consensus.cc:348] T 935f0efc1fe146f4ac21fac4c0cc688c P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:06.674784 7588 raft_consensus.cc:374] T 935f0efc1fe146f4ac21fac4c0cc688c P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:06.674873 7588 raft_consensus.cc:729] T 935f0efc1fe146f4ac21fac4c0cc688c P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Initialized, Role: FOLLOWER
I20240502 20:46:06.675711 7588 consensus_queue.cc:260] T 935f0efc1fe146f4ac21fac4c0cc688c P 201d2b6485084e7dad2cbb722af94779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:06.676075 7588 ts_tablet_manager.cc:1398] T 935f0efc1fe146f4ac21fac4c0cc688c P 201d2b6485084e7dad2cbb722af94779: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:06.676617 7588 tablet_bootstrap.cc:492] T 15a2cbe8285a4938be7f628f7f8b734d P 201d2b6485084e7dad2cbb722af94779: Bootstrap starting.
I20240502 20:46:06.676749 7589 tablet_bootstrap.cc:492] T 15a2cbe8285a4938be7f628f7f8b734d P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:46:06.676903 7589 ts_tablet_manager.cc:1367] T 15a2cbe8285a4938be7f628f7f8b734d P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20240502 20:46:06.677424 7589 raft_consensus.cc:348] T 15a2cbe8285a4938be7f628f7f8b734d P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:06.677618 7589 raft_consensus.cc:374] T 15a2cbe8285a4938be7f628f7f8b734d P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:06.677699 7589 raft_consensus.cc:729] T 15a2cbe8285a4938be7f628f7f8b734d P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:46:06.677858 7588 tablet_bootstrap.cc:654] T 15a2cbe8285a4938be7f628f7f8b734d P 201d2b6485084e7dad2cbb722af94779: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:06.678973 7588 tablet_bootstrap.cc:492] T 15a2cbe8285a4938be7f628f7f8b734d P 201d2b6485084e7dad2cbb722af94779: No bootstrap required, opened a new log
I20240502 20:46:06.679080 7588 ts_tablet_manager.cc:1367] T 15a2cbe8285a4938be7f628f7f8b734d P 201d2b6485084e7dad2cbb722af94779: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:06.677999 7589 consensus_queue.cc:260] T 15a2cbe8285a4938be7f628f7f8b734d P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:06.679525 7588 raft_consensus.cc:348] T 15a2cbe8285a4938be7f628f7f8b734d P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:06.679613 7588 raft_consensus.cc:374] T 15a2cbe8285a4938be7f628f7f8b734d P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:06.679649 7588 raft_consensus.cc:729] T 15a2cbe8285a4938be7f628f7f8b734d P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Initialized, Role: FOLLOWER
I20240502 20:46:06.679787 7588 consensus_queue.cc:260] T 15a2cbe8285a4938be7f628f7f8b734d P 201d2b6485084e7dad2cbb722af94779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:06.680053 7588 ts_tablet_manager.cc:1398] T 15a2cbe8285a4938be7f628f7f8b734d P 201d2b6485084e7dad2cbb722af94779: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:06.681533 7616 ts_tablet_manager.cc:902] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P a4bf1df7fa6b4948be502c182bfbb833: Initiating tablet copy from peer c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097)
I20240502 20:46:06.682341 7616 tablet_copy_client.cc:308] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P a4bf1df7fa6b4948be502c182bfbb833: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.66:42097
I20240502 20:46:06.682518 7589 ts_tablet_manager.cc:1398] T 15a2cbe8285a4938be7f628f7f8b734d P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 20:46:06.683435 4528 tablet_copy_service.cc:140] P c3d26eafd51547acb780d4aac5ebe7cc: Received BeginTabletCopySession request for tablet 61a6d9a8e21e4a098cfad7281f0e0fd6 from peer a4bf1df7fa6b4948be502c182bfbb833 ({username='slave'} at 127.30.32.70:46909)
I20240502 20:46:06.683552 4528 tablet_copy_service.cc:161] P c3d26eafd51547acb780d4aac5ebe7cc: Beginning new tablet copy session on tablet 61a6d9a8e21e4a098cfad7281f0e0fd6 from peer a4bf1df7fa6b4948be502c182bfbb833 at {username='slave'} at 127.30.32.70:46909: session id = a4bf1df7fa6b4948be502c182bfbb833-61a6d9a8e21e4a098cfad7281f0e0fd6
I20240502 20:46:06.685016 4528 tablet_copy_source_session.cc:205] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c3d26eafd51547acb780d4aac5ebe7cc: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:06.686954 7616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61a6d9a8e21e4a098cfad7281f0e0fd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:06.689386 7616 tablet_copy_client.cc:783] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P a4bf1df7fa6b4948be502c182bfbb833: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:06.689574 7616 tablet_copy_client.cc:647] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P a4bf1df7fa6b4948be502c182bfbb833: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:06.691480 7616 tablet_copy_client.cc:523] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P a4bf1df7fa6b4948be502c182bfbb833: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:06.692801 7616 tablet_bootstrap.cc:492] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap starting.
I20240502 20:46:06.711246 4230 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41812:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 20:46:06.717242 6809 tablet_service.cc:1467] Processing CreateTablet for tablet 6f387855819c4af381e9e3d109904081 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e0660e829abf435592bc23d271955609]), partition=
I20240502 20:46:06.717516 6809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f387855819c4af381e9e3d109904081. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:06.717923 6683 tablet_service.cc:1467] Processing CreateTablet for tablet 6f387855819c4af381e9e3d109904081 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e0660e829abf435592bc23d271955609]), partition=
I20240502 20:46:06.718178 6683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f387855819c4af381e9e3d109904081. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:06.719986 7586 tablet_bootstrap.cc:492] T 6f387855819c4af381e9e3d109904081 P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap starting.
I20240502 20:46:06.720896 7586 tablet_bootstrap.cc:654] T 6f387855819c4af381e9e3d109904081 P a4bf1df7fa6b4948be502c182bfbb833: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:06.721683 4749 tablet_service.cc:1467] Processing CreateTablet for tablet 6f387855819c4af381e9e3d109904081 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e0660e829abf435592bc23d271955609]), partition=
I20240502 20:46:06.721930 4749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f387855819c4af381e9e3d109904081. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:06.723963 7587 tablet_bootstrap.cc:492] T 6f387855819c4af381e9e3d109904081 P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:46:06.724771 7587 tablet_bootstrap.cc:654] T 6f387855819c4af381e9e3d109904081 P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:06.725975 7587 tablet_bootstrap.cc:492] T 6f387855819c4af381e9e3d109904081 P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:46:06.726075 7587 ts_tablet_manager.cc:1367] T 6f387855819c4af381e9e3d109904081 P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 20:46:06.726437 7616 tablet_bootstrap.cc:492] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:06.726516 7587 raft_consensus.cc:348] T 6f387855819c4af381e9e3d109904081 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:06.726641 7587 raft_consensus.cc:374] T 6f387855819c4af381e9e3d109904081 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:06.726684 7587 raft_consensus.cc:729] T 6f387855819c4af381e9e3d109904081 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:46:06.726755 7616 tablet_bootstrap.cc:492] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap complete.
I20240502 20:46:06.726823 7587 consensus_queue.cc:260] T 6f387855819c4af381e9e3d109904081 P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:06.726933 7616 ts_tablet_manager.cc:1367] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P a4bf1df7fa6b4948be502c182bfbb833: Time spent bootstrapping tablet: real 0.034s user 0.027s sys 0.003s
I20240502 20:46:06.727066 7168 raft_consensus.cc:482] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:06.727125 7587 ts_tablet_manager.cc:1398] T 6f387855819c4af381e9e3d109904081 P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:06.727190 7586 tablet_bootstrap.cc:492] T 6f387855819c4af381e9e3d109904081 P a4bf1df7fa6b4948be502c182bfbb833: No bootstrap required, opened a new log
I20240502 20:46:06.727270 7586 ts_tablet_manager.cc:1367] T 6f387855819c4af381e9e3d109904081 P a4bf1df7fa6b4948be502c182bfbb833: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 20:46:06.727360 7616 raft_consensus.cc:348] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } }
I20240502 20:46:06.727507 7616 raft_consensus.cc:729] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Initialized, Role: LEARNER
I20240502 20:46:06.727671 7586 raft_consensus.cc:348] T 6f387855819c4af381e9e3d109904081 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:06.727752 7586 raft_consensus.cc:374] T 6f387855819c4af381e9e3d109904081 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:06.727780 7586 raft_consensus.cc:729] T 6f387855819c4af381e9e3d109904081 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Initialized, Role: FOLLOWER
I20240502 20:46:06.727912 7586 consensus_queue.cc:260] T 6f387855819c4af381e9e3d109904081 P a4bf1df7fa6b4948be502c182bfbb833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:06.728159 7586 ts_tablet_manager.cc:1398] T 6f387855819c4af381e9e3d109904081 P a4bf1df7fa6b4948be502c182bfbb833: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:06.727144 7168 raft_consensus.cc:504] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:06.728703 7168 leader_election.cc:288] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
I20240502 20:46:06.727787 7616 consensus_queue.cc:260] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P a4bf1df7fa6b4948be502c182bfbb833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } }
I20240502 20:46:06.729005 7616 ts_tablet_manager.cc:1398] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P a4bf1df7fa6b4948be502c182bfbb833: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 20:46:06.729779 4528 tablet_copy_service.cc:342] P c3d26eafd51547acb780d4aac5ebe7cc: Request end of tablet copy session a4bf1df7fa6b4948be502c182bfbb833-61a6d9a8e21e4a098cfad7281f0e0fd6 received from {username='slave'} at 127.30.32.70:46909
I20240502 20:46:06.729872 4528 tablet_copy_service.cc:433] P c3d26eafd51547acb780d4aac5ebe7cc: ending tablet copy session a4bf1df7fa6b4948be502c182bfbb833-61a6d9a8e21e4a098cfad7281f0e0fd6 on tablet 61a6d9a8e21e4a098cfad7281f0e0fd6 with peer a4bf1df7fa6b4948be502c182bfbb833
I20240502 20:46:06.730729 7642 tablet_bootstrap.cc:492] T 6f387855819c4af381e9e3d109904081 P 7845c397230d49358ba2fad9db53f97c: Bootstrap starting.
I20240502 20:46:06.732734 7642 tablet_bootstrap.cc:654] T 6f387855819c4af381e9e3d109904081 P 7845c397230d49358ba2fad9db53f97c: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:06.733860 7642 tablet_bootstrap.cc:492] T 6f387855819c4af381e9e3d109904081 P 7845c397230d49358ba2fad9db53f97c: No bootstrap required, opened a new log
I20240502 20:46:06.733955 7642 ts_tablet_manager.cc:1367] T 6f387855819c4af381e9e3d109904081 P 7845c397230d49358ba2fad9db53f97c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:46:06.734400 7642 raft_consensus.cc:348] T 6f387855819c4af381e9e3d109904081 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:06.734511 7642 raft_consensus.cc:374] T 6f387855819c4af381e9e3d109904081 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:06.734544 7642 raft_consensus.cc:729] T 6f387855819c4af381e9e3d109904081 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Initialized, Role: FOLLOWER
I20240502 20:46:06.734655 7642 consensus_queue.cc:260] T 6f387855819c4af381e9e3d109904081 P 7845c397230d49358ba2fad9db53f97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:06.734930 7642 ts_tablet_manager.cc:1398] T 6f387855819c4af381e9e3d109904081 P 7845c397230d49358ba2fad9db53f97c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:06.736326 7616 ts_tablet_manager.cc:902] T d37730b109d04b6395f75c7552524877 P a4bf1df7fa6b4948be502c182bfbb833: Initiating tablet copy from peer c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097)
I20240502 20:46:06.736759 7616 tablet_copy_client.cc:308] T d37730b109d04b6395f75c7552524877 P a4bf1df7fa6b4948be502c182bfbb833: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.66:42097
I20240502 20:46:06.737268 4528 tablet_copy_service.cc:140] P c3d26eafd51547acb780d4aac5ebe7cc: Received BeginTabletCopySession request for tablet d37730b109d04b6395f75c7552524877 from peer a4bf1df7fa6b4948be502c182bfbb833 ({username='slave'} at 127.30.32.70:46909)
I20240502 20:46:06.737387 4528 tablet_copy_service.cc:161] P c3d26eafd51547acb780d4aac5ebe7cc: Beginning new tablet copy session on tablet d37730b109d04b6395f75c7552524877 from peer a4bf1df7fa6b4948be502c182bfbb833 at {username='slave'} at 127.30.32.70:46909: session id = a4bf1df7fa6b4948be502c182bfbb833-d37730b109d04b6395f75c7552524877
I20240502 20:46:06.737962 7099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "180902058815460189f75de4704d1e84" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201d2b6485084e7dad2cbb722af94779" is_pre_election: true
I20240502 20:46:06.738093 7099 raft_consensus.cc:2454] T 180902058815460189f75de4704d1e84 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 0.
I20240502 20:46:06.738435 4528 tablet_copy_source_session.cc:205] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:06.738461 6768 leader_election.cc:302] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201d2b6485084e7dad2cbb722af94779, a4bf1df7fa6b4948be502c182bfbb833; no voters:
I20240502 20:46:06.738619 7168 raft_consensus.cc:2781] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:06.738660 7168 raft_consensus.cc:482] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:06.738708 7168 raft_consensus.cc:3037] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:06.739687 7168 raft_consensus.cc:504] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:06.739814 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "180902058815460189f75de4704d1e84" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd" is_pre_election: true
I20240502 20:46:06.739933 4768 raft_consensus.cc:2454] T 180902058815460189f75de4704d1e84 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 0.
I20240502 20:46:06.740051 7168 leader_election.cc:288] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 election: Requested vote from peers ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
I20240502 20:46:06.740386 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "180902058815460189f75de4704d1e84" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd"
I20240502 20:46:06.740448 7099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "180902058815460189f75de4704d1e84" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201d2b6485084e7dad2cbb722af94779"
I20240502 20:46:06.740509 4768 raft_consensus.cc:3037] T 180902058815460189f75de4704d1e84 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:06.740546 7099 raft_consensus.cc:3037] T 180902058815460189f75de4704d1e84 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:06.740703 7616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d37730b109d04b6395f75c7552524877. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:06.741430 7099 raft_consensus.cc:2454] T 180902058815460189f75de4704d1e84 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 1.
I20240502 20:46:06.741430 4768 raft_consensus.cc:2454] T 180902058815460189f75de4704d1e84 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 1.
I20240502 20:46:06.741709 6771 leader_election.cc:302] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4bf1df7fa6b4948be502c182bfbb833, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:46:06.741868 7168 raft_consensus.cc:2781] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:06.741947 7168 raft_consensus.cc:686] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEADER]: Becoming Leader. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Running, Role: LEADER
I20240502 20:46:06.742094 7168 consensus_queue.cc:237] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:06.743582 4235 catalog_manager.cc:5484] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70). New cstate: current_term: 1 leader_uuid: "a4bf1df7fa6b4948be502c182bfbb833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:06.744118 7658 ts_tablet_manager.cc:902] T f352efa74d3c4da3907efdc37237211f P 201d2b6485084e7dad2cbb722af94779: Initiating tablet copy from peer ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:46:06.744319 7616 tablet_copy_client.cc:783] T d37730b109d04b6395f75c7552524877 P a4bf1df7fa6b4948be502c182bfbb833: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:06.744508 7616 tablet_copy_client.cc:647] T d37730b109d04b6395f75c7552524877 P a4bf1df7fa6b4948be502c182bfbb833: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:06.745332 7658 tablet_copy_client.cc:308] T f352efa74d3c4da3907efdc37237211f P 201d2b6485084e7dad2cbb722af94779: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.68:38193
I20240502 20:46:06.745810 7616 tablet_copy_client.cc:523] T d37730b109d04b6395f75c7552524877 P a4bf1df7fa6b4948be502c182bfbb833: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:06.746951 7616 tablet_bootstrap.cc:492] T d37730b109d04b6395f75c7552524877 P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap starting.
I20240502 20:46:06.750233 4790 tablet_copy_service.cc:140] P ed47c023452a497192f35d2f617bf9bd: Received BeginTabletCopySession request for tablet f352efa74d3c4da3907efdc37237211f from peer 201d2b6485084e7dad2cbb722af94779 ({username='slave'} at 127.30.32.72:39947)
I20240502 20:46:06.750362 4790 tablet_copy_service.cc:161] P ed47c023452a497192f35d2f617bf9bd: Beginning new tablet copy session on tablet f352efa74d3c4da3907efdc37237211f from peer 201d2b6485084e7dad2cbb722af94779 at {username='slave'} at 127.30.32.72:39947: session id = 201d2b6485084e7dad2cbb722af94779-f352efa74d3c4da3907efdc37237211f
I20240502 20:46:06.751658 4790 tablet_copy_source_session.cc:205] T f352efa74d3c4da3907efdc37237211f P ed47c023452a497192f35d2f617bf9bd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:06.753499 7658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f352efa74d3c4da3907efdc37237211f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:06.755789 7658 tablet_copy_client.cc:783] T f352efa74d3c4da3907efdc37237211f P 201d2b6485084e7dad2cbb722af94779: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:06.755960 7658 tablet_copy_client.cc:647] T f352efa74d3c4da3907efdc37237211f P 201d2b6485084e7dad2cbb722af94779: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:06.757624 7658 tablet_copy_client.cc:523] T f352efa74d3c4da3907efdc37237211f P 201d2b6485084e7dad2cbb722af94779: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:06.759271 7658 tablet_bootstrap.cc:492] T f352efa74d3c4da3907efdc37237211f P 201d2b6485084e7dad2cbb722af94779: Bootstrap starting.
I20240502 20:46:06.761951 4230 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240502 20:46:06.762075 4230 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240502 20:46:06.764500 4230 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: Sending DeleteTablet for 3 replicas of tablet 0982b98bfed242eba152de225a0158a2
I20240502 20:46:06.764848 4230 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: Sending DeleteTablet for 3 replicas of tablet fa8eb41a63c04e3589ac94ae13a87dc9
I20240502 20:46:06.765166 4230 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: Sending DeleteTablet for 3 replicas of tablet f1f81664a2ee49ef8c775608d3a3d2c2
I20240502 20:46:06.765151 4618 tablet_service.cc:1514] Processing DeleteTablet for tablet 0982b98bfed242eba152de225a0158a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:06 UTC) from {username='slave'} at 127.0.0.1:57068
I20240502 20:46:06.765386 7296 tablet_replica.cc:330] T 0982b98bfed242eba152de225a0158a2 P 923cfab926e7443f8ffd7b21ad0a9d91: stopping tablet replica
I20240502 20:46:06.765564 7296 raft_consensus.cc:2229] T 0982b98bfed242eba152de225a0158a2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:06.765815 7296 raft_consensus.cc:2258] T 0982b98bfed242eba152de225a0158a2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:06.765919 4487 tablet_service.cc:1514] Processing DeleteTablet for tablet fa8eb41a63c04e3589ac94ae13a87dc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:06 UTC) from {username='slave'} at 127.0.0.1:48890
I20240502 20:46:06.766083 7295 tablet_replica.cc:330] T fa8eb41a63c04e3589ac94ae13a87dc9 P c3d26eafd51547acb780d4aac5ebe7cc: stopping tablet replica
I20240502 20:46:06.766180 4488 tablet_service.cc:1514] Processing DeleteTablet for tablet 0982b98bfed242eba152de225a0158a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:06 UTC) from {username='slave'} at 127.0.0.1:48890
I20240502 20:46:06.766358 7296 ts_tablet_manager.cc:1875] T 0982b98bfed242eba152de225a0158a2 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:06.766201 7295 raft_consensus.cc:2229] T fa8eb41a63c04e3589ac94ae13a87dc9 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:06.766525 7295 raft_consensus.cc:2258] T fa8eb41a63c04e3589ac94ae13a87dc9 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:06.766991 7295 ts_tablet_manager.cc:1875] T fa8eb41a63c04e3589ac94ae13a87dc9 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:06.767194 4357 tablet_service.cc:1514] Processing DeleteTablet for tablet 0982b98bfed242eba152de225a0158a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:06 UTC) from {username='slave'} at 127.0.0.1:38306
I20240502 20:46:06.767362 4354 tablet_service.cc:1514] Processing DeleteTablet for tablet fa8eb41a63c04e3589ac94ae13a87dc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:06 UTC) from {username='slave'} at 127.0.0.1:38306
I20240502 20:46:06.767470 4356 tablet_service.cc:1514] Processing DeleteTablet for tablet f1f81664a2ee49ef8c775608d3a3d2c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:06 UTC) from {username='slave'} at 127.0.0.1:38306
I20240502 20:46:06.768541 7296 ts_tablet_manager.cc:1888] T 0982b98bfed242eba152de225a0158a2 P 923cfab926e7443f8ffd7b21ad0a9d91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 20:46:06.768606 7296 log.cc:1196] T 0982b98bfed242eba152de225a0158a2 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal/wals/0982b98bfed242eba152de225a0158a2
I20240502 20:46:06.768893 7296 ts_tablet_manager.cc:1909] T 0982b98bfed242eba152de225a0158a2 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting consensus metadata
I20240502 20:46:06.769343 7295 ts_tablet_manager.cc:1888] T fa8eb41a63c04e3589ac94ae13a87dc9 P c3d26eafd51547acb780d4aac5ebe7cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 20:46:06.769408 7295 log.cc:1196] T fa8eb41a63c04e3589ac94ae13a87dc9 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal/wals/fa8eb41a63c04e3589ac94ae13a87dc9
I20240502 20:46:06.769677 7295 ts_tablet_manager.cc:1909] T fa8eb41a63c04e3589ac94ae13a87dc9 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting consensus metadata
I20240502 20:46:06.770258 7295 tablet_replica.cc:330] T 0982b98bfed242eba152de225a0158a2 P c3d26eafd51547acb780d4aac5ebe7cc: stopping tablet replica
I20240502 20:46:06.770407 7295 raft_consensus.cc:2229] T 0982b98bfed242eba152de225a0158a2 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:06.770526 7295 raft_consensus.cc:2258] T 0982b98bfed242eba152de225a0158a2 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:06.770982 7295 ts_tablet_manager.cc:1875] T 0982b98bfed242eba152de225a0158a2 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:06.770979 4750 tablet_service.cc:1514] Processing DeleteTablet for tablet f1f81664a2ee49ef8c775608d3a3d2c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:06 UTC) from {username='slave'} at 127.0.0.1:38526
I20240502 20:46:06.771224 7299 tablet_replica.cc:330] T f1f81664a2ee49ef8c775608d3a3d2c2 P ed47c023452a497192f35d2f617bf9bd: stopping tablet replica
I20240502 20:46:06.771376 7299 raft_consensus.cc:2229] T f1f81664a2ee49ef8c775608d3a3d2c2 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:06.771526 4618 tablet_service.cc:1514] Processing DeleteTablet for tablet f1f81664a2ee49ef8c775608d3a3d2c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:06 UTC) from {username='slave'} at 127.0.0.1:57068
I20240502 20:46:06.771569 7299 raft_consensus.cc:2258] T f1f81664a2ee49ef8c775608d3a3d2c2 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:06.772063 7299 ts_tablet_manager.cc:1875] T f1f81664a2ee49ef8c775608d3a3d2c2 P ed47c023452a497192f35d2f617bf9bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:06.772215 7296 tablet_replica.cc:330] T f1f81664a2ee49ef8c775608d3a3d2c2 P 923cfab926e7443f8ffd7b21ad0a9d91: stopping tablet replica
I20240502 20:46:06.772336 7296 raft_consensus.cc:2229] T f1f81664a2ee49ef8c775608d3a3d2c2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:06.772367 4219 catalog_manager.cc:4830] TS 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141): tablet 0982b98bfed242eba152de225a0158a2 (table rebalance_test_table_4 [id=129781998e46479d8beddda94bcb5a8f]) successfully deleted
I20240502 20:46:06.772457 7296 raft_consensus.cc:2258] T f1f81664a2ee49ef8c775608d3a3d2c2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:06.772696 4749 tablet_service.cc:1514] Processing DeleteTablet for tablet fa8eb41a63c04e3589ac94ae13a87dc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:06 UTC) from {username='slave'} at 127.0.0.1:38526
I20240502 20:46:06.772884 7297 tablet_replica.cc:330] T 0982b98bfed242eba152de225a0158a2 P c92e6e46782849479c8a0c6d82276fb2: stopping tablet replica
I20240502 20:46:06.773033 7296 ts_tablet_manager.cc:1875] T f1f81664a2ee49ef8c775608d3a3d2c2 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:06.773067 7297 raft_consensus.cc:2229] T 0982b98bfed242eba152de225a0158a2 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:06.773226 7297 raft_consensus.cc:2258] T 0982b98bfed242eba152de225a0158a2 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:06.773778 7297 ts_tablet_manager.cc:1875] T 0982b98bfed242eba152de225a0158a2 P c92e6e46782849479c8a0c6d82276fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:06.774279 7299 ts_tablet_manager.cc:1888] T f1f81664a2ee49ef8c775608d3a3d2c2 P ed47c023452a497192f35d2f617bf9bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 20:46:06.774343 7299 log.cc:1196] T f1f81664a2ee49ef8c775608d3a3d2c2 P ed47c023452a497192f35d2f617bf9bd: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal/wals/f1f81664a2ee49ef8c775608d3a3d2c2
I20240502 20:46:06.774715 4221 catalog_manager.cc:4830] TS c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097): tablet fa8eb41a63c04e3589ac94ae13a87dc9 (table rebalance_test_table_4 [id=129781998e46479d8beddda94bcb5a8f]) successfully deleted
I20240502 20:46:06.775163 7295 ts_tablet_manager.cc:1888] T 0982b98bfed242eba152de225a0158a2 P c3d26eafd51547acb780d4aac5ebe7cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 20:46:06.775252 7295 log.cc:1196] T 0982b98bfed242eba152de225a0158a2 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal/wals/0982b98bfed242eba152de225a0158a2
I20240502 20:46:06.775290 7296 ts_tablet_manager.cc:1888] T f1f81664a2ee49ef8c775608d3a3d2c2 P 923cfab926e7443f8ffd7b21ad0a9d91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 20:46:06.775355 7296 log.cc:1196] T f1f81664a2ee49ef8c775608d3a3d2c2 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal/wals/f1f81664a2ee49ef8c775608d3a3d2c2
I20240502 20:46:06.775552 7295 ts_tablet_manager.cc:1909] T 0982b98bfed242eba152de225a0158a2 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting consensus metadata
I20240502 20:46:06.775616 7296 ts_tablet_manager.cc:1909] T f1f81664a2ee49ef8c775608d3a3d2c2 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting consensus metadata
I20240502 20:46:06.776062 7297 ts_tablet_manager.cc:1888] T 0982b98bfed242eba152de225a0158a2 P c92e6e46782849479c8a0c6d82276fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 20:46:06.776144 7297 log.cc:1196] T 0982b98bfed242eba152de225a0158a2 P c92e6e46782849479c8a0c6d82276fb2: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal/wals/0982b98bfed242eba152de225a0158a2
I20240502 20:46:06.776422 7297 ts_tablet_manager.cc:1909] T 0982b98bfed242eba152de225a0158a2 P c92e6e46782849479c8a0c6d82276fb2: Deleting consensus metadata
I20240502 20:46:06.776422 7322 raft_consensus.cc:482] T 935f0efc1fe146f4ac21fac4c0cc688c P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:06.776528 7322 raft_consensus.cc:504] T 935f0efc1fe146f4ac21fac4c0cc688c P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:06.777035 7322 leader_election.cc:288] T 935f0efc1fe146f4ac21fac4c0cc688c P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
I20240502 20:46:06.777246 7297 tablet_replica.cc:330] T fa8eb41a63c04e3589ac94ae13a87dc9 P c92e6e46782849479c8a0c6d82276fb2: stopping tablet replica
I20240502 20:46:06.777313 4222 catalog_manager.cc:4830] TS c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827): tablet 0982b98bfed242eba152de225a0158a2 (table rebalance_test_table_4 [id=129781998e46479d8beddda94bcb5a8f]) successfully deleted
I20240502 20:46:06.777390 7297 raft_consensus.cc:2229] T fa8eb41a63c04e3589ac94ae13a87dc9 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:06.777525 7297 raft_consensus.cc:2258] T fa8eb41a63c04e3589ac94ae13a87dc9 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:06.777555 7299 ts_tablet_manager.cc:1909] T f1f81664a2ee49ef8c775608d3a3d2c2 P ed47c023452a497192f35d2f617bf9bd: Deleting consensus metadata
I20240502 20:46:06.777951 4221 catalog_manager.cc:4830] TS c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097): tablet 0982b98bfed242eba152de225a0158a2 (table rebalance_test_table_4 [id=129781998e46479d8beddda94bcb5a8f]) successfully deleted
I20240502 20:46:06.777944 7099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "935f0efc1fe146f4ac21fac4c0cc688c" candidate_uuid: "a47e5c56f13843018ac584914a8d4707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201d2b6485084e7dad2cbb722af94779" is_pre_election: true
I20240502 20:46:06.778087 7099 raft_consensus.cc:2454] T 935f0efc1fe146f4ac21fac4c0cc688c P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a47e5c56f13843018ac584914a8d4707 in term 0.
I20240502 20:46:06.778458 7297 ts_tablet_manager.cc:1875] T fa8eb41a63c04e3589ac94ae13a87dc9 P c92e6e46782849479c8a0c6d82276fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:06.778474 6900 leader_election.cc:302] T 935f0efc1fe146f4ac21fac4c0cc688c P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201d2b6485084e7dad2cbb722af94779, a47e5c56f13843018ac584914a8d4707; no voters:
I20240502 20:46:06.780879 7297 ts_tablet_manager.cc:1888] T fa8eb41a63c04e3589ac94ae13a87dc9 P c92e6e46782849479c8a0c6d82276fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 20:46:06.780946 7297 log.cc:1196] T fa8eb41a63c04e3589ac94ae13a87dc9 P c92e6e46782849479c8a0c6d82276fb2: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal/wals/fa8eb41a63c04e3589ac94ae13a87dc9
I20240502 20:46:06.781235 7297 ts_tablet_manager.cc:1909] T fa8eb41a63c04e3589ac94ae13a87dc9 P c92e6e46782849479c8a0c6d82276fb2: Deleting consensus metadata
I20240502 20:46:06.782091 7322 raft_consensus.cc:2781] T 935f0efc1fe146f4ac21fac4c0cc688c P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:06.782151 7322 raft_consensus.cc:482] T 935f0efc1fe146f4ac21fac4c0cc688c P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:06.782188 7322 raft_consensus.cc:3037] T 935f0efc1fe146f4ac21fac4c0cc688c P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:06.783113 7322 raft_consensus.cc:504] T 935f0efc1fe146f4ac21fac4c0cc688c P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:06.783489 7322 leader_election.cc:288] T 935f0efc1fe146f4ac21fac4c0cc688c P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 1 election: Requested vote from peers c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
I20240502 20:46:06.783845 6375 raft_consensus.cc:482] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:06.783932 6375 raft_consensus.cc:504] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:06.783946 7099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "935f0efc1fe146f4ac21fac4c0cc688c" candidate_uuid: "a47e5c56f13843018ac584914a8d4707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201d2b6485084e7dad2cbb722af94779"
I20240502 20:46:06.784065 7099 raft_consensus.cc:3037] T 935f0efc1fe146f4ac21fac4c0cc688c P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:06.784308 6375 leader_election.cc:288] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
I20240502 20:46:06.784808 4219 catalog_manager.cc:4830] TS 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141): tablet f1f81664a2ee49ef8c775608d3a3d2c2 (table rebalance_test_table_4 [id=129781998e46479d8beddda94bcb5a8f]) successfully deleted
I20240502 20:46:06.785105 6957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "935f0efc1fe146f4ac21fac4c0cc688c" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47e5c56f13843018ac584914a8d4707" is_pre_election: true
I20240502 20:46:06.785272 6957 raft_consensus.cc:2379] T 935f0efc1fe146f4ac21fac4c0cc688c P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c3d26eafd51547acb780d4aac5ebe7cc in current term 1: Already voted for candidate a47e5c56f13843018ac584914a8d4707 in this term.
I20240502 20:46:06.784971 7099 raft_consensus.cc:2454] T 935f0efc1fe146f4ac21fac4c0cc688c P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a47e5c56f13843018ac584914a8d4707 in term 1.
I20240502 20:46:06.785837 7098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "935f0efc1fe146f4ac21fac4c0cc688c" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201d2b6485084e7dad2cbb722af94779" is_pre_election: true
I20240502 20:46:06.786593 7098 raft_consensus.cc:2379] T 935f0efc1fe146f4ac21fac4c0cc688c P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c3d26eafd51547acb780d4aac5ebe7cc in current term 1: Already voted for candidate a47e5c56f13843018ac584914a8d4707 in this term.
I20240502 20:46:06.786728 7297 tablet_replica.cc:330] T f1f81664a2ee49ef8c775608d3a3d2c2 P c92e6e46782849479c8a0c6d82276fb2: stopping tablet replica
I20240502 20:46:06.786867 7297 raft_consensus.cc:2229] T f1f81664a2ee49ef8c775608d3a3d2c2 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:06.786932 4441 leader_election.cc:302] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3d26eafd51547acb780d4aac5ebe7cc; no voters: 201d2b6485084e7dad2cbb722af94779, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:06.786981 7297 raft_consensus.cc:2258] T f1f81664a2ee49ef8c775608d3a3d2c2 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:06.787158 6375 raft_consensus.cc:3037] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:06.787220 7616 tablet_bootstrap.cc:492] T d37730b109d04b6395f75c7552524877 P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:06.787475 7616 tablet_bootstrap.cc:492] T d37730b109d04b6395f75c7552524877 P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap complete.
I20240502 20:46:06.787501 7297 ts_tablet_manager.cc:1875] T f1f81664a2ee49ef8c775608d3a3d2c2 P c92e6e46782849479c8a0c6d82276fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:06.787629 7616 ts_tablet_manager.cc:1367] T d37730b109d04b6395f75c7552524877 P a4bf1df7fa6b4948be502c182bfbb833: Time spent bootstrapping tablet: real 0.041s user 0.028s sys 0.000s
I20240502 20:46:06.788108 6375 raft_consensus.cc:2726] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 20:46:06.788141 7616 raft_consensus.cc:348] T d37730b109d04b6395f75c7552524877 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } }
I20240502 20:46:06.788265 7616 raft_consensus.cc:729] T d37730b109d04b6395f75c7552524877 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Initialized, Role: LEARNER
I20240502 20:46:06.788388 7616 consensus_queue.cc:260] T d37730b109d04b6395f75c7552524877 P a4bf1df7fa6b4948be502c182bfbb833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } }
I20240502 20:46:06.788623 4506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "935f0efc1fe146f4ac21fac4c0cc688c" candidate_uuid: "a47e5c56f13843018ac584914a8d4707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" is_pre_election: true
I20240502 20:46:06.788682 4222 catalog_manager.cc:4830] TS c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827): tablet fa8eb41a63c04e3589ac94ae13a87dc9 (table rebalance_test_table_4 [id=129781998e46479d8beddda94bcb5a8f]) successfully deleted
I20240502 20:46:06.788743 4506 raft_consensus.cc:2454] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a47e5c56f13843018ac584914a8d4707 in term 1.
I20240502 20:46:06.788935 4508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "935f0efc1fe146f4ac21fac4c0cc688c" candidate_uuid: "a47e5c56f13843018ac584914a8d4707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc"
I20240502 20:46:06.789346 7616 ts_tablet_manager.cc:1398] T d37730b109d04b6395f75c7552524877 P a4bf1df7fa6b4948be502c182bfbb833: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:06.789330 6900 leader_election.cc:302] T 935f0efc1fe146f4ac21fac4c0cc688c P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201d2b6485084e7dad2cbb722af94779, a47e5c56f13843018ac584914a8d4707; no voters:
I20240502 20:46:06.789500 7310 raft_consensus.cc:2781] T 935f0efc1fe146f4ac21fac4c0cc688c P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:06.789580 7310 raft_consensus.cc:686] T 935f0efc1fe146f4ac21fac4c0cc688c P a47e5c56f13843018ac584914a8d4707 [term 1 LEADER]: Becoming Leader. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Running, Role: LEADER
I20240502 20:46:06.789726 7297 ts_tablet_manager.cc:1888] T f1f81664a2ee49ef8c775608d3a3d2c2 P c92e6e46782849479c8a0c6d82276fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 20:46:06.789788 7297 log.cc:1196] T f1f81664a2ee49ef8c775608d3a3d2c2 P c92e6e46782849479c8a0c6d82276fb2: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal/wals/f1f81664a2ee49ef8c775608d3a3d2c2
I20240502 20:46:06.789714 7310 consensus_queue.cc:237] T 935f0efc1fe146f4ac21fac4c0cc688c P a47e5c56f13843018ac584914a8d4707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:06.789863 4508 raft_consensus.cc:2454] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a47e5c56f13843018ac584914a8d4707 in term 1.
I20240502 20:46:06.790048 7297 ts_tablet_manager.cc:1909] T f1f81664a2ee49ef8c775608d3a3d2c2 P c92e6e46782849479c8a0c6d82276fb2: Deleting consensus metadata
I20240502 20:46:06.790359 4528 tablet_copy_service.cc:342] P c3d26eafd51547acb780d4aac5ebe7cc: Request end of tablet copy session a4bf1df7fa6b4948be502c182bfbb833-d37730b109d04b6395f75c7552524877 received from {username='slave'} at 127.30.32.70:46909
I20240502 20:46:06.790426 4528 tablet_copy_service.cc:433] P c3d26eafd51547acb780d4aac5ebe7cc: ending tablet copy session a4bf1df7fa6b4948be502c182bfbb833-d37730b109d04b6395f75c7552524877 on tablet d37730b109d04b6395f75c7552524877 with peer a4bf1df7fa6b4948be502c182bfbb833
I20240502 20:46:06.793414 4222 catalog_manager.cc:4830] TS c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827): tablet f1f81664a2ee49ef8c775608d3a3d2c2 (table rebalance_test_table_4 [id=129781998e46479d8beddda94bcb5a8f]) successfully deleted
I20240502 20:46:06.793491 7685 ts_tablet_manager.cc:902] T f6d4a3019ba54eb8b3506b88f04a6262 P a47e5c56f13843018ac584914a8d4707: Initiating tablet copy from peer c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:46:06.794222 7685 tablet_copy_client.cc:308] T f6d4a3019ba54eb8b3506b88f04a6262 P a47e5c56f13843018ac584914a8d4707: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.65:33827
I20240502 20:46:06.794955 7690 ts_tablet_manager.cc:902] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 201d2b6485084e7dad2cbb722af94779: Initiating tablet copy from peer c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097)
I20240502 20:46:06.794986 4397 tablet_copy_service.cc:140] P c92e6e46782849479c8a0c6d82276fb2: Received BeginTabletCopySession request for tablet f6d4a3019ba54eb8b3506b88f04a6262 from peer a47e5c56f13843018ac584914a8d4707 ({username='slave'} at 127.30.32.71:33061)
I20240502 20:46:06.795107 4397 tablet_copy_service.cc:161] P c92e6e46782849479c8a0c6d82276fb2: Beginning new tablet copy session on tablet f6d4a3019ba54eb8b3506b88f04a6262 from peer a47e5c56f13843018ac584914a8d4707 at {username='slave'} at 127.30.32.71:33061: session id = a47e5c56f13843018ac584914a8d4707-f6d4a3019ba54eb8b3506b88f04a6262
I20240502 20:46:06.795961 7690 tablet_copy_client.cc:308] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 201d2b6485084e7dad2cbb722af94779: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.66:42097
I20240502 20:46:06.796365 4235 catalog_manager.cc:5484] T 935f0efc1fe146f4ac21fac4c0cc688c P a47e5c56f13843018ac584914a8d4707 reported cstate change: term changed from 0 to 1, leader changed from <none> to a47e5c56f13843018ac584914a8d4707 (127.30.32.71). New cstate: current_term: 1 leader_uuid: "a47e5c56f13843018ac584914a8d4707" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: HEALTHY } } }
I20240502 20:46:06.796519 4397 tablet_copy_source_session.cc:205] T f6d4a3019ba54eb8b3506b88f04a6262 P c92e6e46782849479c8a0c6d82276fb2: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:06.798928 4222 catalog_manager.cc:4830] TS ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193): tablet f1f81664a2ee49ef8c775608d3a3d2c2 (table rebalance_test_table_4 [id=129781998e46479d8beddda94bcb5a8f]) successfully deleted
I20240502 20:46:06.799160 7299 tablet_replica.cc:330] T fa8eb41a63c04e3589ac94ae13a87dc9 P ed47c023452a497192f35d2f617bf9bd: stopping tablet replica
I20240502 20:46:06.799284 7299 raft_consensus.cc:2229] T fa8eb41a63c04e3589ac94ae13a87dc9 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:06.799481 7299 raft_consensus.cc:2258] T fa8eb41a63c04e3589ac94ae13a87dc9 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:06.799953 7299 ts_tablet_manager.cc:1875] T fa8eb41a63c04e3589ac94ae13a87dc9 P ed47c023452a497192f35d2f617bf9bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:06.800271 7685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6d4a3019ba54eb8b3506b88f04a6262. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:06.801288 4528 tablet_copy_service.cc:140] P c3d26eafd51547acb780d4aac5ebe7cc: Received BeginTabletCopySession request for tablet 5a36bf6e8e6e43b2b47b6d6357b0d445 from peer 201d2b6485084e7dad2cbb722af94779 ({username='slave'} at 127.30.32.72:54159)
I20240502 20:46:06.801385 4528 tablet_copy_service.cc:161] P c3d26eafd51547acb780d4aac5ebe7cc: Beginning new tablet copy session on tablet 5a36bf6e8e6e43b2b47b6d6357b0d445 from peer 201d2b6485084e7dad2cbb722af94779 at {username='slave'} at 127.30.32.72:54159: session id = 201d2b6485084e7dad2cbb722af94779-5a36bf6e8e6e43b2b47b6d6357b0d445
I20240502 20:46:06.802322 4528 tablet_copy_source_session.cc:205] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:06.802699 7685 tablet_copy_client.cc:783] T f6d4a3019ba54eb8b3506b88f04a6262 P a47e5c56f13843018ac584914a8d4707: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:06.802876 7685 tablet_copy_client.cc:647] T f6d4a3019ba54eb8b3506b88f04a6262 P a47e5c56f13843018ac584914a8d4707: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:06.803532 7690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a36bf6e8e6e43b2b47b6d6357b0d445. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:06.804317 7299 ts_tablet_manager.cc:1888] T fa8eb41a63c04e3589ac94ae13a87dc9 P ed47c023452a497192f35d2f617bf9bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 20:46:06.804792 7685 tablet_copy_client.cc:523] T f6d4a3019ba54eb8b3506b88f04a6262 P a47e5c56f13843018ac584914a8d4707: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:06.805722 7299 log.cc:1196] T fa8eb41a63c04e3589ac94ae13a87dc9 P ed47c023452a497192f35d2f617bf9bd: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal/wals/fa8eb41a63c04e3589ac94ae13a87dc9
I20240502 20:46:06.806092 7299 ts_tablet_manager.cc:1909] T fa8eb41a63c04e3589ac94ae13a87dc9 P ed47c023452a497192f35d2f617bf9bd: Deleting consensus metadata
I20240502 20:46:06.806159 7690 tablet_copy_client.cc:783] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 201d2b6485084e7dad2cbb722af94779: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:06.806345 7685 tablet_bootstrap.cc:492] T f6d4a3019ba54eb8b3506b88f04a6262 P a47e5c56f13843018ac584914a8d4707: Bootstrap starting.
I20240502 20:46:06.806416 7690 tablet_copy_client.cc:647] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 201d2b6485084e7dad2cbb722af94779: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:06.809021 4222 catalog_manager.cc:4830] TS ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193): tablet fa8eb41a63c04e3589ac94ae13a87dc9 (table rebalance_test_table_4 [id=129781998e46479d8beddda94bcb5a8f]) successfully deleted
I20240502 20:46:06.810019 7682 ts_tablet_manager.cc:902] T a2e81d2c63cd4e4cac5086506b5c4a12 P 7845c397230d49358ba2fad9db53f97c: Initiating tablet copy from peer ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:46:06.810142 7690 tablet_copy_client.cc:523] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 201d2b6485084e7dad2cbb722af94779: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:06.811079 7682 tablet_copy_client.cc:308] T a2e81d2c63cd4e4cac5086506b5c4a12 P 7845c397230d49358ba2fad9db53f97c: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.68:38193
I20240502 20:46:06.811607 7690 tablet_bootstrap.cc:492] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 201d2b6485084e7dad2cbb722af94779: Bootstrap starting.
I20240502 20:46:06.812736 7658 tablet_bootstrap.cc:492] T f352efa74d3c4da3907efdc37237211f P 201d2b6485084e7dad2cbb722af94779: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:06.813150 7658 tablet_bootstrap.cc:492] T f352efa74d3c4da3907efdc37237211f P 201d2b6485084e7dad2cbb722af94779: Bootstrap complete.
I20240502 20:46:06.813413 7658 ts_tablet_manager.cc:1367] T f352efa74d3c4da3907efdc37237211f P 201d2b6485084e7dad2cbb722af94779: Time spent bootstrapping tablet: real 0.054s user 0.034s sys 0.002s
I20240502 20:46:06.813923 7658 raft_consensus.cc:348] T f352efa74d3c4da3907efdc37237211f P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } }
I20240502 20:46:06.814078 7658 raft_consensus.cc:729] T f352efa74d3c4da3907efdc37237211f P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Initialized, Role: LEARNER
I20240502 20:46:06.814250 7658 consensus_queue.cc:260] T f352efa74d3c4da3907efdc37237211f P 201d2b6485084e7dad2cbb722af94779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } }
I20240502 20:46:06.814563 7658 ts_tablet_manager.cc:1398] T f352efa74d3c4da3907efdc37237211f P 201d2b6485084e7dad2cbb722af94779: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:06.815193 4790 tablet_copy_service.cc:342] P ed47c023452a497192f35d2f617bf9bd: Request end of tablet copy session 201d2b6485084e7dad2cbb722af94779-f352efa74d3c4da3907efdc37237211f received from {username='slave'} at 127.30.32.72:39947
I20240502 20:46:06.815282 4790 tablet_copy_service.cc:433] P ed47c023452a497192f35d2f617bf9bd: ending tablet copy session 201d2b6485084e7dad2cbb722af94779-f352efa74d3c4da3907efdc37237211f on tablet f352efa74d3c4da3907efdc37237211f with peer 201d2b6485084e7dad2cbb722af94779
I20240502 20:46:06.818756 4790 tablet_copy_service.cc:140] P ed47c023452a497192f35d2f617bf9bd: Received BeginTabletCopySession request for tablet a2e81d2c63cd4e4cac5086506b5c4a12 from peer 7845c397230d49358ba2fad9db53f97c ({username='slave'} at 127.30.32.69:43573)
I20240502 20:46:06.818862 4790 tablet_copy_service.cc:161] P ed47c023452a497192f35d2f617bf9bd: Beginning new tablet copy session on tablet a2e81d2c63cd4e4cac5086506b5c4a12 from peer 7845c397230d49358ba2fad9db53f97c at {username='slave'} at 127.30.32.69:43573: session id = 7845c397230d49358ba2fad9db53f97c-a2e81d2c63cd4e4cac5086506b5c4a12
I20240502 20:46:06.819824 4790 tablet_copy_source_session.cc:205] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:06.820495 7695 ts_tablet_manager.cc:902] T a62a7ce526f14c25994bb9e0e3c3bc84 P 7845c397230d49358ba2fad9db53f97c: Initiating tablet copy from peer c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097)
I20240502 20:46:06.821130 7695 tablet_copy_client.cc:308] T a62a7ce526f14c25994bb9e0e3c3bc84 P 7845c397230d49358ba2fad9db53f97c: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.66:42097
I20240502 20:46:06.823609 7682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2e81d2c63cd4e4cac5086506b5c4a12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:06.825174 4528 tablet_copy_service.cc:140] P c3d26eafd51547acb780d4aac5ebe7cc: Received BeginTabletCopySession request for tablet a62a7ce526f14c25994bb9e0e3c3bc84 from peer 7845c397230d49358ba2fad9db53f97c ({username='slave'} at 127.30.32.69:48755)
I20240502 20:46:06.825284 4528 tablet_copy_service.cc:161] P c3d26eafd51547acb780d4aac5ebe7cc: Beginning new tablet copy session on tablet a62a7ce526f14c25994bb9e0e3c3bc84 from peer 7845c397230d49358ba2fad9db53f97c at {username='slave'} at 127.30.32.69:48755: session id = 7845c397230d49358ba2fad9db53f97c-a62a7ce526f14c25994bb9e0e3c3bc84
I20240502 20:46:06.826318 4528 tablet_copy_source_session.cc:205] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:06.827164 7695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a62a7ce526f14c25994bb9e0e3c3bc84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:06.829370 7682 tablet_copy_client.cc:783] T a2e81d2c63cd4e4cac5086506b5c4a12 P 7845c397230d49358ba2fad9db53f97c: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:06.829624 7695 tablet_copy_client.cc:783] T a62a7ce526f14c25994bb9e0e3c3bc84 P 7845c397230d49358ba2fad9db53f97c: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:06.830132 7682 tablet_copy_client.cc:647] T a2e81d2c63cd4e4cac5086506b5c4a12 P 7845c397230d49358ba2fad9db53f97c: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:06.830286 7695 tablet_copy_client.cc:647] T a62a7ce526f14c25994bb9e0e3c3bc84 P 7845c397230d49358ba2fad9db53f97c: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:06.831621 7695 tablet_copy_client.cc:523] T a62a7ce526f14c25994bb9e0e3c3bc84 P 7845c397230d49358ba2fad9db53f97c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:06.831621 7682 tablet_copy_client.cc:523] T a2e81d2c63cd4e4cac5086506b5c4a12 P 7845c397230d49358ba2fad9db53f97c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:06.833370 7682 tablet_bootstrap.cc:492] T a2e81d2c63cd4e4cac5086506b5c4a12 P 7845c397230d49358ba2fad9db53f97c: Bootstrap starting.
I20240502 20:46:06.833413 7695 tablet_bootstrap.cc:492] T a62a7ce526f14c25994bb9e0e3c3bc84 P 7845c397230d49358ba2fad9db53f97c: Bootstrap starting.
I20240502 20:46:06.847095 7690 tablet_bootstrap.cc:492] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 201d2b6485084e7dad2cbb722af94779: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:06.847414 7690 tablet_bootstrap.cc:492] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 201d2b6485084e7dad2cbb722af94779: Bootstrap complete.
I20240502 20:46:06.847607 7690 ts_tablet_manager.cc:1367] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 201d2b6485084e7dad2cbb722af94779: Time spent bootstrapping tablet: real 0.036s user 0.024s sys 0.001s
I20240502 20:46:06.848066 7690 raft_consensus.cc:348] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } }
I20240502 20:46:06.848244 7690 raft_consensus.cc:729] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Initialized, Role: LEARNER
I20240502 20:46:06.848431 7690 consensus_queue.cc:260] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 201d2b6485084e7dad2cbb722af94779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } }
I20240502 20:46:06.848762 7690 ts_tablet_manager.cc:1398] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 201d2b6485084e7dad2cbb722af94779: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240502 20:46:06.849535 4528 tablet_copy_service.cc:342] P c3d26eafd51547acb780d4aac5ebe7cc: Request end of tablet copy session 201d2b6485084e7dad2cbb722af94779-5a36bf6e8e6e43b2b47b6d6357b0d445 received from {username='slave'} at 127.30.32.72:54159
I20240502 20:46:06.849627 4528 tablet_copy_service.cc:433] P c3d26eafd51547acb780d4aac5ebe7cc: ending tablet copy session 201d2b6485084e7dad2cbb722af94779-5a36bf6e8e6e43b2b47b6d6357b0d445 on tablet 5a36bf6e8e6e43b2b47b6d6357b0d445 with peer 201d2b6485084e7dad2cbb722af94779
I20240502 20:46:06.860670 7168 raft_consensus.cc:482] T 6f387855819c4af381e9e3d109904081 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:06.860795 7168 raft_consensus.cc:504] T 6f387855819c4af381e9e3d109904081 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:06.861202 7168 leader_election.cc:288] T 6f387855819c4af381e9e3d109904081 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:46:06.861917 6702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f387855819c4af381e9e3d109904081" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7845c397230d49358ba2fad9db53f97c" is_pre_election: true
I20240502 20:46:06.862066 6702 raft_consensus.cc:2454] T 6f387855819c4af381e9e3d109904081 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 0.
I20240502 20:46:06.862067 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f387855819c4af381e9e3d109904081" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd" is_pre_election: true
I20240502 20:46:06.862205 4768 raft_consensus.cc:2454] T 6f387855819c4af381e9e3d109904081 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 0.
I20240502 20:46:06.862493 6771 leader_election.cc:302] T 6f387855819c4af381e9e3d109904081 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4bf1df7fa6b4948be502c182bfbb833, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:46:06.862656 7168 raft_consensus.cc:2781] T 6f387855819c4af381e9e3d109904081 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:06.862707 7168 raft_consensus.cc:482] T 6f387855819c4af381e9e3d109904081 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:06.862751 7168 raft_consensus.cc:3037] T 6f387855819c4af381e9e3d109904081 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:06.863704 7168 raft_consensus.cc:504] T 6f387855819c4af381e9e3d109904081 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:06.864058 7168 leader_election.cc:288] T 6f387855819c4af381e9e3d109904081 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 election: Requested vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:46:06.864751 6702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f387855819c4af381e9e3d109904081" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7845c397230d49358ba2fad9db53f97c"
I20240502 20:46:06.864799 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f387855819c4af381e9e3d109904081" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd"
I20240502 20:46:06.864864 6702 raft_consensus.cc:3037] T 6f387855819c4af381e9e3d109904081 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:06.864893 4768 raft_consensus.cc:3037] T 6f387855819c4af381e9e3d109904081 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:06.865835 4768 raft_consensus.cc:2454] T 6f387855819c4af381e9e3d109904081 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 1.
I20240502 20:46:06.865849 6702 raft_consensus.cc:2454] T 6f387855819c4af381e9e3d109904081 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 1.
I20240502 20:46:06.866266 6770 leader_election.cc:302] T 6f387855819c4af381e9e3d109904081 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7845c397230d49358ba2fad9db53f97c, a4bf1df7fa6b4948be502c182bfbb833; no voters:
I20240502 20:46:06.866526 7168 raft_consensus.cc:2781] T 6f387855819c4af381e9e3d109904081 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:06.866591 7168 raft_consensus.cc:686] T 6f387855819c4af381e9e3d109904081 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEADER]: Becoming Leader. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Running, Role: LEADER
I20240502 20:46:06.866699 7168 consensus_queue.cc:237] T 6f387855819c4af381e9e3d109904081 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:06.869480 4235 catalog_manager.cc:5484] T 6f387855819c4af381e9e3d109904081 P a4bf1df7fa6b4948be502c182bfbb833 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70). New cstate: current_term: 1 leader_uuid: "a4bf1df7fa6b4948be502c182bfbb833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:06.880703 7685 tablet_bootstrap.cc:492] T f6d4a3019ba54eb8b3506b88f04a6262 P a47e5c56f13843018ac584914a8d4707: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:06.881067 7685 tablet_bootstrap.cc:492] T f6d4a3019ba54eb8b3506b88f04a6262 P a47e5c56f13843018ac584914a8d4707: Bootstrap complete.
I20240502 20:46:06.882436 7682 tablet_bootstrap.cc:492] T a2e81d2c63cd4e4cac5086506b5c4a12 P 7845c397230d49358ba2fad9db53f97c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:06.883296 7695 tablet_bootstrap.cc:492] T a62a7ce526f14c25994bb9e0e3c3bc84 P 7845c397230d49358ba2fad9db53f97c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:06.883558 7685 ts_tablet_manager.cc:1367] T f6d4a3019ba54eb8b3506b88f04a6262 P a47e5c56f13843018ac584914a8d4707: Time spent bootstrapping tablet: real 0.077s user 0.031s sys 0.004s
I20240502 20:46:06.883589 7695 tablet_bootstrap.cc:492] T a62a7ce526f14c25994bb9e0e3c3bc84 P 7845c397230d49358ba2fad9db53f97c: Bootstrap complete.
I20240502 20:46:06.883751 7695 ts_tablet_manager.cc:1367] T a62a7ce526f14c25994bb9e0e3c3bc84 P 7845c397230d49358ba2fad9db53f97c: Time spent bootstrapping tablet: real 0.050s user 0.026s sys 0.001s
I20240502 20:46:06.883984 7682 tablet_bootstrap.cc:492] T a2e81d2c63cd4e4cac5086506b5c4a12 P 7845c397230d49358ba2fad9db53f97c: Bootstrap complete.
I20240502 20:46:06.884186 7682 ts_tablet_manager.cc:1367] T a2e81d2c63cd4e4cac5086506b5c4a12 P 7845c397230d49358ba2fad9db53f97c: Time spent bootstrapping tablet: real 0.051s user 0.028s sys 0.000s
I20240502 20:46:06.884141 7695 raft_consensus.cc:348] T a62a7ce526f14c25994bb9e0e3c3bc84 P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } }
I20240502 20:46:06.884266 7695 raft_consensus.cc:729] T a62a7ce526f14c25994bb9e0e3c3bc84 P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Initialized, Role: LEARNER
I20240502 20:46:06.884141 7685 raft_consensus.cc:348] T f6d4a3019ba54eb8b3506b88f04a6262 P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } }
I20240502 20:46:06.884577 7682 raft_consensus.cc:348] T a2e81d2c63cd4e4cac5086506b5c4a12 P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } }
I20240502 20:46:06.884706 7682 raft_consensus.cc:729] T a2e81d2c63cd4e4cac5086506b5c4a12 P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Initialized, Role: LEARNER
I20240502 20:46:06.884689 7685 raft_consensus.cc:729] T f6d4a3019ba54eb8b3506b88f04a6262 P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Initialized, Role: LEARNER
I20240502 20:46:06.884812 7682 consensus_queue.cc:260] T a2e81d2c63cd4e4cac5086506b5c4a12 P 7845c397230d49358ba2fad9db53f97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } }
I20240502 20:46:06.884657 7695 consensus_queue.cc:260] T a62a7ce526f14c25994bb9e0e3c3bc84 P 7845c397230d49358ba2fad9db53f97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } }
I20240502 20:46:06.884930 7685 consensus_queue.cc:260] T f6d4a3019ba54eb8b3506b88f04a6262 P a47e5c56f13843018ac584914a8d4707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } }
I20240502 20:46:06.885222 7695 ts_tablet_manager.cc:1398] T a62a7ce526f14c25994bb9e0e3c3bc84 P 7845c397230d49358ba2fad9db53f97c: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240502 20:46:06.885520 7685 ts_tablet_manager.cc:1398] T f6d4a3019ba54eb8b3506b88f04a6262 P a47e5c56f13843018ac584914a8d4707: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:06.886099 4528 tablet_copy_service.cc:342] P c3d26eafd51547acb780d4aac5ebe7cc: Request end of tablet copy session 7845c397230d49358ba2fad9db53f97c-a62a7ce526f14c25994bb9e0e3c3bc84 received from {username='slave'} at 127.30.32.69:48755
I20240502 20:46:06.886184 4528 tablet_copy_service.cc:433] P c3d26eafd51547acb780d4aac5ebe7cc: ending tablet copy session 7845c397230d49358ba2fad9db53f97c-a62a7ce526f14c25994bb9e0e3c3bc84 on tablet a62a7ce526f14c25994bb9e0e3c3bc84 with peer 7845c397230d49358ba2fad9db53f97c
I20240502 20:46:06.887272 4397 tablet_copy_service.cc:342] P c92e6e46782849479c8a0c6d82276fb2: Request end of tablet copy session a47e5c56f13843018ac584914a8d4707-f6d4a3019ba54eb8b3506b88f04a6262 received from {username='slave'} at 127.30.32.71:33061
I20240502 20:46:06.887382 4397 tablet_copy_service.cc:433] P c92e6e46782849479c8a0c6d82276fb2: ending tablet copy session a47e5c56f13843018ac584914a8d4707-f6d4a3019ba54eb8b3506b88f04a6262 on tablet f6d4a3019ba54eb8b3506b88f04a6262 with peer a47e5c56f13843018ac584914a8d4707
I20240502 20:46:06.889199 7685 ts_tablet_manager.cc:902] T 47c5087093214fbeae35590b4b3c3d42 P a47e5c56f13843018ac584914a8d4707: Initiating tablet copy from peer c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:46:06.889598 7685 tablet_copy_client.cc:308] T 47c5087093214fbeae35590b4b3c3d42 P a47e5c56f13843018ac584914a8d4707: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.65:33827
I20240502 20:46:06.890405 4397 tablet_copy_service.cc:140] P c92e6e46782849479c8a0c6d82276fb2: Received BeginTabletCopySession request for tablet 47c5087093214fbeae35590b4b3c3d42 from peer a47e5c56f13843018ac584914a8d4707 ({username='slave'} at 127.30.32.71:33061)
I20240502 20:46:06.890516 4397 tablet_copy_service.cc:161] P c92e6e46782849479c8a0c6d82276fb2: Beginning new tablet copy session on tablet 47c5087093214fbeae35590b4b3c3d42 from peer a47e5c56f13843018ac584914a8d4707 at {username='slave'} at 127.30.32.71:33061: session id = a47e5c56f13843018ac584914a8d4707-47c5087093214fbeae35590b4b3c3d42
I20240502 20:46:06.891542 4397 tablet_copy_source_session.cc:205] T 47c5087093214fbeae35590b4b3c3d42 P c92e6e46782849479c8a0c6d82276fb2: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:06.892382 7685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47c5087093214fbeae35590b4b3c3d42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:06.893761 7682 ts_tablet_manager.cc:1398] T a2e81d2c63cd4e4cac5086506b5c4a12 P 7845c397230d49358ba2fad9db53f97c: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20240502 20:46:06.894587 7685 tablet_copy_client.cc:783] T 47c5087093214fbeae35590b4b3c3d42 P a47e5c56f13843018ac584914a8d4707: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:06.894783 7685 tablet_copy_client.cc:647] T 47c5087093214fbeae35590b4b3c3d42 P a47e5c56f13843018ac584914a8d4707: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:06.894871 4790 tablet_copy_service.cc:342] P ed47c023452a497192f35d2f617bf9bd: Request end of tablet copy session 7845c397230d49358ba2fad9db53f97c-a2e81d2c63cd4e4cac5086506b5c4a12 received from {username='slave'} at 127.30.32.69:43573
I20240502 20:46:06.894958 4790 tablet_copy_service.cc:433] P ed47c023452a497192f35d2f617bf9bd: ending tablet copy session 7845c397230d49358ba2fad9db53f97c-a2e81d2c63cd4e4cac5086506b5c4a12 on tablet a2e81d2c63cd4e4cac5086506b5c4a12 with peer 7845c397230d49358ba2fad9db53f97c
I20240502 20:46:06.898895 7685 tablet_copy_client.cc:523] T 47c5087093214fbeae35590b4b3c3d42 P a47e5c56f13843018ac584914a8d4707: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:06.900110 7685 tablet_bootstrap.cc:492] T 47c5087093214fbeae35590b4b3c3d42 P a47e5c56f13843018ac584914a8d4707: Bootstrap starting.
I20240502 20:46:06.910465 7616 ts_tablet_manager.cc:902] T 7abfd61f55554042978b50761cbd6ab5 P a4bf1df7fa6b4948be502c182bfbb833: Initiating tablet copy from peer 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:46:06.910984 7616 tablet_copy_client.cc:308] T 7abfd61f55554042978b50761cbd6ab5 P a4bf1df7fa6b4948be502c182bfbb833: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.67:42141
I20240502 20:46:06.916417 4659 tablet_copy_service.cc:140] P 923cfab926e7443f8ffd7b21ad0a9d91: Received BeginTabletCopySession request for tablet 7abfd61f55554042978b50761cbd6ab5 from peer a4bf1df7fa6b4948be502c182bfbb833 ({username='slave'} at 127.30.32.70:32901)
I20240502 20:46:06.916570 4659 tablet_copy_service.cc:161] P 923cfab926e7443f8ffd7b21ad0a9d91: Beginning new tablet copy session on tablet 7abfd61f55554042978b50761cbd6ab5 from peer a4bf1df7fa6b4948be502c182bfbb833 at {username='slave'} at 127.30.32.70:32901: session id = a4bf1df7fa6b4948be502c182bfbb833-7abfd61f55554042978b50761cbd6ab5
I20240502 20:46:06.918027 4659 tablet_copy_source_session.cc:205] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:06.919335 7616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7abfd61f55554042978b50761cbd6ab5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:06.921432 7616 tablet_copy_client.cc:783] T 7abfd61f55554042978b50761cbd6ab5 P a4bf1df7fa6b4948be502c182bfbb833: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:06.921615 7616 tablet_copy_client.cc:647] T 7abfd61f55554042978b50761cbd6ab5 P a4bf1df7fa6b4948be502c182bfbb833: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:06.924481 7616 tablet_copy_client.cc:523] T 7abfd61f55554042978b50761cbd6ab5 P a4bf1df7fa6b4948be502c182bfbb833: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:06.925814 7616 tablet_bootstrap.cc:492] T 7abfd61f55554042978b50761cbd6ab5 P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap starting.
I20240502 20:46:06.930958 7685 tablet_bootstrap.cc:492] T 47c5087093214fbeae35590b4b3c3d42 P a47e5c56f13843018ac584914a8d4707: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:06.931324 7685 tablet_bootstrap.cc:492] T 47c5087093214fbeae35590b4b3c3d42 P a47e5c56f13843018ac584914a8d4707: Bootstrap complete.
I20240502 20:46:06.931519 7685 ts_tablet_manager.cc:1367] T 47c5087093214fbeae35590b4b3c3d42 P a47e5c56f13843018ac584914a8d4707: Time spent bootstrapping tablet: real 0.031s user 0.025s sys 0.000s
I20240502 20:46:06.931972 7685 raft_consensus.cc:348] T 47c5087093214fbeae35590b4b3c3d42 P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } }
I20240502 20:46:06.932175 7685 raft_consensus.cc:729] T 47c5087093214fbeae35590b4b3c3d42 P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Initialized, Role: LEARNER
I20240502 20:46:06.932297 7685 consensus_queue.cc:260] T 47c5087093214fbeae35590b4b3c3d42 P a47e5c56f13843018ac584914a8d4707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } }
I20240502 20:46:06.932538 7685 ts_tablet_manager.cc:1398] T 47c5087093214fbeae35590b4b3c3d42 P a47e5c56f13843018ac584914a8d4707: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:06.933256 4397 tablet_copy_service.cc:342] P c92e6e46782849479c8a0c6d82276fb2: Request end of tablet copy session a47e5c56f13843018ac584914a8d4707-47c5087093214fbeae35590b4b3c3d42 received from {username='slave'} at 127.30.32.71:33061
I20240502 20:46:06.933344 4397 tablet_copy_service.cc:433] P c92e6e46782849479c8a0c6d82276fb2: ending tablet copy session a47e5c56f13843018ac584914a8d4707-47c5087093214fbeae35590b4b3c3d42 on tablet 47c5087093214fbeae35590b4b3c3d42 with peer a47e5c56f13843018ac584914a8d4707
I20240502 20:46:06.945781 7616 tablet_bootstrap.cc:492] T 7abfd61f55554042978b50761cbd6ab5 P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:06.946019 7616 tablet_bootstrap.cc:492] T 7abfd61f55554042978b50761cbd6ab5 P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap complete.
I20240502 20:46:06.946157 7616 ts_tablet_manager.cc:1367] T 7abfd61f55554042978b50761cbd6ab5 P a4bf1df7fa6b4948be502c182bfbb833: Time spent bootstrapping tablet: real 0.020s user 0.019s sys 0.000s
I20240502 20:46:06.946496 7616 raft_consensus.cc:348] T 7abfd61f55554042978b50761cbd6ab5 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } }
I20240502 20:46:06.946590 7616 raft_consensus.cc:729] T 7abfd61f55554042978b50761cbd6ab5 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Initialized, Role: LEARNER
I20240502 20:46:06.946674 7616 consensus_queue.cc:260] T 7abfd61f55554042978b50761cbd6ab5 P a4bf1df7fa6b4948be502c182bfbb833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } }
I20240502 20:46:06.946870 7616 ts_tablet_manager.cc:1398] T 7abfd61f55554042978b50761cbd6ab5 P a4bf1df7fa6b4948be502c182bfbb833: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:06.947613 4659 tablet_copy_service.cc:342] P 923cfab926e7443f8ffd7b21ad0a9d91: Request end of tablet copy session a4bf1df7fa6b4948be502c182bfbb833-7abfd61f55554042978b50761cbd6ab5 received from {username='slave'} at 127.30.32.70:32901
I20240502 20:46:06.947717 4659 tablet_copy_service.cc:433] P 923cfab926e7443f8ffd7b21ad0a9d91: ending tablet copy session a4bf1df7fa6b4948be502c182bfbb833-7abfd61f55554042978b50761cbd6ab5 on tablet 7abfd61f55554042978b50761cbd6ab5 with peer a4bf1df7fa6b4948be502c182bfbb833
I20240502 20:46:06.956781 7682 ts_tablet_manager.cc:902] T d6fb394fdcff436886fe0634d79c024c P 7845c397230d49358ba2fad9db53f97c: Initiating tablet copy from peer c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097)
I20240502 20:46:06.956912 7690 ts_tablet_manager.cc:902] T 9e34e8e17a7e413b97eb614a7df727d5 P 201d2b6485084e7dad2cbb722af94779: Initiating tablet copy from peer ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:46:06.957283 7682 tablet_copy_client.cc:308] T d6fb394fdcff436886fe0634d79c024c P 7845c397230d49358ba2fad9db53f97c: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.66:42097
I20240502 20:46:06.957403 7690 tablet_copy_client.cc:308] T 9e34e8e17a7e413b97eb614a7df727d5 P 201d2b6485084e7dad2cbb722af94779: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.68:38193
I20240502 20:46:06.957847 4790 tablet_copy_service.cc:140] P ed47c023452a497192f35d2f617bf9bd: Received BeginTabletCopySession request for tablet 9e34e8e17a7e413b97eb614a7df727d5 from peer 201d2b6485084e7dad2cbb722af94779 ({username='slave'} at 127.30.32.72:39947)
I20240502 20:46:06.957856 4528 tablet_copy_service.cc:140] P c3d26eafd51547acb780d4aac5ebe7cc: Received BeginTabletCopySession request for tablet d6fb394fdcff436886fe0634d79c024c from peer 7845c397230d49358ba2fad9db53f97c ({username='slave'} at 127.30.32.69:48755)
I20240502 20:46:06.957943 4790 tablet_copy_service.cc:161] P ed47c023452a497192f35d2f617bf9bd: Beginning new tablet copy session on tablet 9e34e8e17a7e413b97eb614a7df727d5 from peer 201d2b6485084e7dad2cbb722af94779 at {username='slave'} at 127.30.32.72:39947: session id = 201d2b6485084e7dad2cbb722af94779-9e34e8e17a7e413b97eb614a7df727d5
I20240502 20:46:06.957947 4528 tablet_copy_service.cc:161] P c3d26eafd51547acb780d4aac5ebe7cc: Beginning new tablet copy session on tablet d6fb394fdcff436886fe0634d79c024c from peer 7845c397230d49358ba2fad9db53f97c at {username='slave'} at 127.30.32.69:48755: session id = 7845c397230d49358ba2fad9db53f97c-d6fb394fdcff436886fe0634d79c024c
I20240502 20:46:06.958611 4528 tablet_copy_source_session.cc:205] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:06.958659 4790 tablet_copy_source_session.cc:205] T 9e34e8e17a7e413b97eb614a7df727d5 P ed47c023452a497192f35d2f617bf9bd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:06.959391 7682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6fb394fdcff436886fe0634d79c024c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:06.959391 7690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e34e8e17a7e413b97eb614a7df727d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:06.961066 7682 tablet_copy_client.cc:783] T d6fb394fdcff436886fe0634d79c024c P 7845c397230d49358ba2fad9db53f97c: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:06.961161 7690 tablet_copy_client.cc:783] T 9e34e8e17a7e413b97eb614a7df727d5 P 201d2b6485084e7dad2cbb722af94779: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:06.961218 7682 tablet_copy_client.cc:647] T d6fb394fdcff436886fe0634d79c024c P 7845c397230d49358ba2fad9db53f97c: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:06.961311 7690 tablet_copy_client.cc:647] T 9e34e8e17a7e413b97eb614a7df727d5 P 201d2b6485084e7dad2cbb722af94779: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:06.962524 7690 tablet_copy_client.cc:523] T 9e34e8e17a7e413b97eb614a7df727d5 P 201d2b6485084e7dad2cbb722af94779: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:06.962602 7682 tablet_copy_client.cc:523] T d6fb394fdcff436886fe0634d79c024c P 7845c397230d49358ba2fad9db53f97c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:06.963456 7690 tablet_bootstrap.cc:492] T 9e34e8e17a7e413b97eb614a7df727d5 P 201d2b6485084e7dad2cbb722af94779: Bootstrap starting.
I20240502 20:46:06.963480 7682 tablet_bootstrap.cc:492] T d6fb394fdcff436886fe0634d79c024c P 7845c397230d49358ba2fad9db53f97c: Bootstrap starting.
I20240502 20:46:06.980537 7685 ts_tablet_manager.cc:902] T 8565ebbd21dd4f708a0254d1ed2d6670 P a47e5c56f13843018ac584914a8d4707: Initiating tablet copy from peer 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:46:06.981045 7685 tablet_copy_client.cc:308] T 8565ebbd21dd4f708a0254d1ed2d6670 P a47e5c56f13843018ac584914a8d4707: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.67:42141
I20240502 20:46:06.983448 7682 tablet_bootstrap.cc:492] T d6fb394fdcff436886fe0634d79c024c P 7845c397230d49358ba2fad9db53f97c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:06.983719 7690 tablet_bootstrap.cc:492] T 9e34e8e17a7e413b97eb614a7df727d5 P 201d2b6485084e7dad2cbb722af94779: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:06.983747 7682 tablet_bootstrap.cc:492] T d6fb394fdcff436886fe0634d79c024c P 7845c397230d49358ba2fad9db53f97c: Bootstrap complete.
I20240502 20:46:06.983916 7682 ts_tablet_manager.cc:1367] T d6fb394fdcff436886fe0634d79c024c P 7845c397230d49358ba2fad9db53f97c: Time spent bootstrapping tablet: real 0.020s user 0.019s sys 0.000s
I20240502 20:46:06.983989 7690 tablet_bootstrap.cc:492] T 9e34e8e17a7e413b97eb614a7df727d5 P 201d2b6485084e7dad2cbb722af94779: Bootstrap complete.
I20240502 20:46:06.984172 7690 ts_tablet_manager.cc:1367] T 9e34e8e17a7e413b97eb614a7df727d5 P 201d2b6485084e7dad2cbb722af94779: Time spent bootstrapping tablet: real 0.021s user 0.018s sys 0.000s
I20240502 20:46:06.984390 7682 raft_consensus.cc:348] T d6fb394fdcff436886fe0634d79c024c P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } }
I20240502 20:46:06.984537 7682 raft_consensus.cc:729] T d6fb394fdcff436886fe0634d79c024c P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Initialized, Role: LEARNER
I20240502 20:46:06.984679 7682 consensus_queue.cc:260] T d6fb394fdcff436886fe0634d79c024c P 7845c397230d49358ba2fad9db53f97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } }
I20240502 20:46:06.984786 7690 raft_consensus.cc:348] T 9e34e8e17a7e413b97eb614a7df727d5 P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } }
I20240502 20:46:06.984998 7690 raft_consensus.cc:729] T 9e34e8e17a7e413b97eb614a7df727d5 P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Initialized, Role: LEARNER
I20240502 20:46:06.985078 7682 ts_tablet_manager.cc:1398] T d6fb394fdcff436886fe0634d79c024c P 7845c397230d49358ba2fad9db53f97c: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240502 20:46:06.985198 7690 consensus_queue.cc:260] T 9e34e8e17a7e413b97eb614a7df727d5 P 201d2b6485084e7dad2cbb722af94779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } }
I20240502 20:46:06.985553 7690 ts_tablet_manager.cc:1398] T 9e34e8e17a7e413b97eb614a7df727d5 P 201d2b6485084e7dad2cbb722af94779: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:06.986049 4528 tablet_copy_service.cc:342] P c3d26eafd51547acb780d4aac5ebe7cc: Request end of tablet copy session 7845c397230d49358ba2fad9db53f97c-d6fb394fdcff436886fe0634d79c024c received from {username='slave'} at 127.30.32.69:48755
I20240502 20:46:06.986076 4790 tablet_copy_service.cc:342] P ed47c023452a497192f35d2f617bf9bd: Request end of tablet copy session 201d2b6485084e7dad2cbb722af94779-9e34e8e17a7e413b97eb614a7df727d5 received from {username='slave'} at 127.30.32.72:39947
I20240502 20:46:06.986130 4528 tablet_copy_service.cc:433] P c3d26eafd51547acb780d4aac5ebe7cc: ending tablet copy session 7845c397230d49358ba2fad9db53f97c-d6fb394fdcff436886fe0634d79c024c on tablet d6fb394fdcff436886fe0634d79c024c with peer 7845c397230d49358ba2fad9db53f97c
I20240502 20:46:06.986140 4790 tablet_copy_service.cc:433] P ed47c023452a497192f35d2f617bf9bd: ending tablet copy session 201d2b6485084e7dad2cbb722af94779-9e34e8e17a7e413b97eb614a7df727d5 on tablet 9e34e8e17a7e413b97eb614a7df727d5 with peer 201d2b6485084e7dad2cbb722af94779
I20240502 20:46:06.987107 4659 tablet_copy_service.cc:140] P 923cfab926e7443f8ffd7b21ad0a9d91: Received BeginTabletCopySession request for tablet 8565ebbd21dd4f708a0254d1ed2d6670 from peer a47e5c56f13843018ac584914a8d4707 ({username='slave'} at 127.30.32.71:59213)
I20240502 20:46:06.987210 4659 tablet_copy_service.cc:161] P 923cfab926e7443f8ffd7b21ad0a9d91: Beginning new tablet copy session on tablet 8565ebbd21dd4f708a0254d1ed2d6670 from peer a47e5c56f13843018ac584914a8d4707 at {username='slave'} at 127.30.32.71:59213: session id = a47e5c56f13843018ac584914a8d4707-8565ebbd21dd4f708a0254d1ed2d6670
I20240502 20:46:06.988113 4659 tablet_copy_source_session.cc:205] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:06.988451 7695 ts_tablet_manager.cc:902] T 00eb91e624554baf918e96b098019be8 P 7845c397230d49358ba2fad9db53f97c: Initiating tablet copy from peer 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:46:06.988994 7695 tablet_copy_client.cc:308] T 00eb91e624554baf918e96b098019be8 P 7845c397230d49358ba2fad9db53f97c: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.67:42141
I20240502 20:46:06.990295 7685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8565ebbd21dd4f708a0254d1ed2d6670. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:06.992558 7685 tablet_copy_client.cc:783] T 8565ebbd21dd4f708a0254d1ed2d6670 P a47e5c56f13843018ac584914a8d4707: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:06.992728 7685 tablet_copy_client.cc:647] T 8565ebbd21dd4f708a0254d1ed2d6670 P a47e5c56f13843018ac584914a8d4707: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:06.993295 7690 ts_tablet_manager.cc:902] T 28029d4e66d1453cbde9811ffc2a3283 P 201d2b6485084e7dad2cbb722af94779: Initiating tablet copy from peer c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097)
I20240502 20:46:06.993736 7690 tablet_copy_client.cc:308] T 28029d4e66d1453cbde9811ffc2a3283 P 201d2b6485084e7dad2cbb722af94779: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.66:42097
I20240502 20:46:06.993764 4659 tablet_copy_service.cc:140] P 923cfab926e7443f8ffd7b21ad0a9d91: Received BeginTabletCopySession request for tablet 00eb91e624554baf918e96b098019be8 from peer 7845c397230d49358ba2fad9db53f97c ({username='slave'} at 127.30.32.69:34201)
I20240502 20:46:06.993841 4659 tablet_copy_service.cc:161] P 923cfab926e7443f8ffd7b21ad0a9d91: Beginning new tablet copy session on tablet 00eb91e624554baf918e96b098019be8 from peer 7845c397230d49358ba2fad9db53f97c at {username='slave'} at 127.30.32.69:34201: session id = 7845c397230d49358ba2fad9db53f97c-00eb91e624554baf918e96b098019be8
I20240502 20:46:06.994550 4528 tablet_copy_service.cc:140] P c3d26eafd51547acb780d4aac5ebe7cc: Received BeginTabletCopySession request for tablet 28029d4e66d1453cbde9811ffc2a3283 from peer 201d2b6485084e7dad2cbb722af94779 ({username='slave'} at 127.30.32.72:54159)
I20240502 20:46:06.994570 4659 tablet_copy_source_session.cc:205] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:06.994649 4528 tablet_copy_service.cc:161] P c3d26eafd51547acb780d4aac5ebe7cc: Beginning new tablet copy session on tablet 28029d4e66d1453cbde9811ffc2a3283 from peer 201d2b6485084e7dad2cbb722af94779 at {username='slave'} at 127.30.32.72:54159: session id = 201d2b6485084e7dad2cbb722af94779-28029d4e66d1453cbde9811ffc2a3283
I20240502 20:46:06.995458 7695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00eb91e624554baf918e96b098019be8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:06.995543 4528 tablet_copy_source_session.cc:205] T 28029d4e66d1453cbde9811ffc2a3283 P c3d26eafd51547acb780d4aac5ebe7cc: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:06.996240 7690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28029d4e66d1453cbde9811ffc2a3283. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:06.997260 7695 tablet_copy_client.cc:783] T 00eb91e624554baf918e96b098019be8 P 7845c397230d49358ba2fad9db53f97c: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:06.997346 7685 tablet_copy_client.cc:523] T 8565ebbd21dd4f708a0254d1ed2d6670 P a47e5c56f13843018ac584914a8d4707: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:06.997406 7695 tablet_copy_client.cc:647] T 00eb91e624554baf918e96b098019be8 P 7845c397230d49358ba2fad9db53f97c: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:06.997970 4235 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 20:46:06.998562 7685 tablet_bootstrap.cc:492] T 8565ebbd21dd4f708a0254d1ed2d6670 P a47e5c56f13843018ac584914a8d4707: Bootstrap starting.
I20240502 20:46:06.998581 7690 tablet_copy_client.cc:783] T 28029d4e66d1453cbde9811ffc2a3283 P 201d2b6485084e7dad2cbb722af94779: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:06.998809 7690 tablet_copy_client.cc:647] T 28029d4e66d1453cbde9811ffc2a3283 P 201d2b6485084e7dad2cbb722af94779: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:07.000166 7690 tablet_copy_client.cc:523] T 28029d4e66d1453cbde9811ffc2a3283 P 201d2b6485084e7dad2cbb722af94779: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:07.001446 7695 tablet_copy_client.cc:523] T 00eb91e624554baf918e96b098019be8 P 7845c397230d49358ba2fad9db53f97c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:07.001708 7690 tablet_bootstrap.cc:492] T 28029d4e66d1453cbde9811ffc2a3283 P 201d2b6485084e7dad2cbb722af94779: Bootstrap starting.
I20240502 20:46:07.002909 4768 raft_consensus.cc:1261] T 6f387855819c4af381e9e3d109904081 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer a4bf1df7fa6b4948be502c182bfbb833: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:07.003347 7401 consensus_queue.cc:1035] T 6f387855819c4af381e9e3d109904081 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:07.003677 6702 raft_consensus.cc:1261] T 6f387855819c4af381e9e3d109904081 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Refusing update from remote peer a4bf1df7fa6b4948be502c182bfbb833: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:07.004909 7401 consensus_queue.cc:1035] T 6f387855819c4af381e9e3d109904081 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:07.007325 4230 catalog_manager.cc:5576] TS 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429) does not have the latest schema for tablet 6f387855819c4af381e9e3d109904081 (table rebalancer_dynamic_table [id=e0660e829abf435592bc23d271955609]). Expected version 1 got 0
I20240502 20:46:07.008128 7695 tablet_bootstrap.cc:492] T 00eb91e624554baf918e96b098019be8 P 7845c397230d49358ba2fad9db53f97c: Bootstrap starting.
I20240502 20:46:07.009006 4230 catalog_manager.cc:5576] TS ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193) does not have the latest schema for tablet 6f387855819c4af381e9e3d109904081 (table rebalancer_dynamic_table [id=e0660e829abf435592bc23d271955609]). Expected version 1 got 0
I20240502 20:46:07.013523 4230 catalog_manager.cc:5576] TS a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867) does not have the latest schema for tablet 6f387855819c4af381e9e3d109904081 (table rebalancer_dynamic_table [id=e0660e829abf435592bc23d271955609]). Expected version 1 got 0
I20240502 20:46:07.016281 4235 catalog_manager.cc:5576] TS 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429) does not have the latest schema for tablet 6f387855819c4af381e9e3d109904081 (table rebalancer_dynamic_table [id=e0660e829abf435592bc23d271955609]). Expected version 1 got 0
I20240502 20:46:07.018007 7790 tablet.cc:1722] T 6f387855819c4af381e9e3d109904081 P 7845c397230d49358ba2fad9db53f97c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 20:46:07.018384 7790 tablet.cc:1620] T 6f387855819c4af381e9e3d109904081 P 7845c397230d49358ba2fad9db53f97c: MemRowSet was empty: no flush needed.
I20240502 20:46:07.018611 7789 tablet.cc:1722] T 6f387855819c4af381e9e3d109904081 P a4bf1df7fa6b4948be502c182bfbb833: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 20:46:07.019981 7789 tablet.cc:1620] T 6f387855819c4af381e9e3d109904081 P a4bf1df7fa6b4948be502c182bfbb833: MemRowSet was empty: no flush needed.
I20240502 20:46:07.028092 7685 tablet_bootstrap.cc:492] T 8565ebbd21dd4f708a0254d1ed2d6670 P a47e5c56f13843018ac584914a8d4707: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:07.028402 7685 tablet_bootstrap.cc:492] T 8565ebbd21dd4f708a0254d1ed2d6670 P a47e5c56f13843018ac584914a8d4707: Bootstrap complete.
I20240502 20:46:07.028436 4235 catalog_manager.cc:5576] TS ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193) does not have the latest schema for tablet 6f387855819c4af381e9e3d109904081 (table rebalancer_dynamic_table [id=e0660e829abf435592bc23d271955609]). Expected version 1 got 0
I20240502 20:46:07.028580 7685 ts_tablet_manager.cc:1367] T 8565ebbd21dd4f708a0254d1ed2d6670 P a47e5c56f13843018ac584914a8d4707: Time spent bootstrapping tablet: real 0.030s user 0.027s sys 0.000s
I20240502 20:46:07.029105 7685 raft_consensus.cc:348] T 8565ebbd21dd4f708a0254d1ed2d6670 P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } }
I20240502 20:46:07.029244 7685 raft_consensus.cc:729] T 8565ebbd21dd4f708a0254d1ed2d6670 P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Initialized, Role: LEARNER
I20240502 20:46:07.029353 7685 consensus_queue.cc:260] T 8565ebbd21dd4f708a0254d1ed2d6670 P a47e5c56f13843018ac584914a8d4707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } }
I20240502 20:46:07.029759 4230 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: rebalancer_dynamic_table [id=e0660e829abf435592bc23d271955609] alter complete (version 1)
I20240502 20:46:07.031694 7685 ts_tablet_manager.cc:1398] T 8565ebbd21dd4f708a0254d1ed2d6670 P a47e5c56f13843018ac584914a8d4707: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:46:07.034025 4658 tablet_copy_service.cc:342] P 923cfab926e7443f8ffd7b21ad0a9d91: Request end of tablet copy session a47e5c56f13843018ac584914a8d4707-8565ebbd21dd4f708a0254d1ed2d6670 received from {username='slave'} at 127.30.32.71:59213
I20240502 20:46:07.034114 4658 tablet_copy_service.cc:433] P 923cfab926e7443f8ffd7b21ad0a9d91: ending tablet copy session a47e5c56f13843018ac584914a8d4707-8565ebbd21dd4f708a0254d1ed2d6670 on tablet 8565ebbd21dd4f708a0254d1ed2d6670 with peer a47e5c56f13843018ac584914a8d4707
I20240502 20:46:07.035516 7432 raft_consensus.cc:482] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:07.035630 7432 raft_consensus.cc:504] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:07.036020 7432 leader_election.cc:288] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
I20240502 20:46:07.036381 4508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15a2cbe8285a4938be7f628f7f8b734d" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" is_pre_election: true
I20240502 20:46:07.036499 4508 raft_consensus.cc:2454] T 15a2cbe8285a4938be7f628f7f8b734d P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 0.
I20240502 20:46:07.036518 7098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15a2cbe8285a4938be7f628f7f8b734d" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201d2b6485084e7dad2cbb722af94779" is_pre_election: true
I20240502 20:46:07.036659 7098 raft_consensus.cc:2454] T 15a2cbe8285a4938be7f628f7f8b734d P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 0.
I20240502 20:46:07.036761 4312 leader_election.cc:302] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3d26eafd51547acb780d4aac5ebe7cc, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:46:07.036954 7432 raft_consensus.cc:2781] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:07.037017 7432 raft_consensus.cc:482] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:07.037055 7432 raft_consensus.cc:3037] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:07.038065 7432 raft_consensus.cc:504] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:07.038457 7432 leader_election.cc:288] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 election: Requested vote from peers c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
I20240502 20:46:07.038837 4508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15a2cbe8285a4938be7f628f7f8b734d" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc"
I20240502 20:46:07.038861 7098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15a2cbe8285a4938be7f628f7f8b734d" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201d2b6485084e7dad2cbb722af94779"
I20240502 20:46:07.038959 7098 raft_consensus.cc:3037] T 15a2cbe8285a4938be7f628f7f8b734d P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:07.038959 4508 raft_consensus.cc:3037] T 15a2cbe8285a4938be7f628f7f8b734d P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:07.039945 7098 raft_consensus.cc:2454] T 15a2cbe8285a4938be7f628f7f8b734d P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 1.
I20240502 20:46:07.039945 4508 raft_consensus.cc:2454] T 15a2cbe8285a4938be7f628f7f8b734d P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 1.
I20240502 20:46:07.040264 4312 leader_election.cc:302] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3d26eafd51547acb780d4aac5ebe7cc, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:46:07.040439 7432 raft_consensus.cc:2781] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:07.040531 7432 raft_consensus.cc:686] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEADER]: Becoming Leader. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Running, Role: LEADER
I20240502 20:46:07.040681 7432 consensus_queue.cc:237] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:07.041225 7791 tablet.cc:1722] T 6f387855819c4af381e9e3d109904081 P ed47c023452a497192f35d2f617bf9bd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 20:46:07.042220 7791 tablet.cc:1620] T 6f387855819c4af381e9e3d109904081 P ed47c023452a497192f35d2f617bf9bd: MemRowSet was empty: no flush needed.
I20240502 20:46:07.042965 4235 catalog_manager.cc:5484] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65). New cstate: current_term: 1 leader_uuid: "c92e6e46782849479c8a0c6d82276fb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:07.051945 7695 tablet_bootstrap.cc:492] T 00eb91e624554baf918e96b098019be8 P 7845c397230d49358ba2fad9db53f97c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:07.052029 7690 tablet_bootstrap.cc:492] T 28029d4e66d1453cbde9811ffc2a3283 P 201d2b6485084e7dad2cbb722af94779: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:07.052274 7695 tablet_bootstrap.cc:492] T 00eb91e624554baf918e96b098019be8 P 7845c397230d49358ba2fad9db53f97c: Bootstrap complete.
I20240502 20:46:07.052273 7690 tablet_bootstrap.cc:492] T 28029d4e66d1453cbde9811ffc2a3283 P 201d2b6485084e7dad2cbb722af94779: Bootstrap complete.
I20240502 20:46:07.052444 7690 ts_tablet_manager.cc:1367] T 28029d4e66d1453cbde9811ffc2a3283 P 201d2b6485084e7dad2cbb722af94779: Time spent bootstrapping tablet: real 0.051s user 0.021s sys 0.004s
I20240502 20:46:07.052444 7695 ts_tablet_manager.cc:1367] T 00eb91e624554baf918e96b098019be8 P 7845c397230d49358ba2fad9db53f97c: Time spent bootstrapping tablet: real 0.044s user 0.021s sys 0.004s
I20240502 20:46:07.052922 7695 raft_consensus.cc:348] T 00eb91e624554baf918e96b098019be8 P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } }
I20240502 20:46:07.052923 7690 raft_consensus.cc:348] T 28029d4e66d1453cbde9811ffc2a3283 P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } }
I20240502 20:46:07.053068 7690 raft_consensus.cc:729] T 28029d4e66d1453cbde9811ffc2a3283 P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Initialized, Role: LEARNER
I20240502 20:46:07.053069 7695 raft_consensus.cc:729] T 00eb91e624554baf918e96b098019be8 P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Initialized, Role: LEARNER
I20240502 20:46:07.053179 7690 consensus_queue.cc:260] T 28029d4e66d1453cbde9811ffc2a3283 P 201d2b6485084e7dad2cbb722af94779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } }
I20240502 20:46:07.053179 7695 consensus_queue.cc:260] T 00eb91e624554baf918e96b098019be8 P 7845c397230d49358ba2fad9db53f97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } }
I20240502 20:46:07.053580 7695 ts_tablet_manager.cc:1398] T 00eb91e624554baf918e96b098019be8 P 7845c397230d49358ba2fad9db53f97c: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240502 20:46:07.054157 4658 tablet_copy_service.cc:342] P 923cfab926e7443f8ffd7b21ad0a9d91: Request end of tablet copy session 7845c397230d49358ba2fad9db53f97c-00eb91e624554baf918e96b098019be8 received from {username='slave'} at 127.30.32.69:34201
I20240502 20:46:07.054244 4658 tablet_copy_service.cc:433] P 923cfab926e7443f8ffd7b21ad0a9d91: ending tablet copy session 7845c397230d49358ba2fad9db53f97c-00eb91e624554baf918e96b098019be8 on tablet 00eb91e624554baf918e96b098019be8 with peer 7845c397230d49358ba2fad9db53f97c
I20240502 20:46:07.059442 7616 ts_tablet_manager.cc:902] T 02bddc1866e04858894e047ec100a6f0 P a4bf1df7fa6b4948be502c182bfbb833: Initiating tablet copy from peer ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:46:07.059955 7616 tablet_copy_client.cc:308] T 02bddc1866e04858894e047ec100a6f0 P a4bf1df7fa6b4948be502c182bfbb833: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.68:38193
I20240502 20:46:07.060468 4790 tablet_copy_service.cc:140] P ed47c023452a497192f35d2f617bf9bd: Received BeginTabletCopySession request for tablet 02bddc1866e04858894e047ec100a6f0 from peer a4bf1df7fa6b4948be502c182bfbb833 ({username='slave'} at 127.30.32.70:40425)
I20240502 20:46:07.060586 4790 tablet_copy_service.cc:161] P ed47c023452a497192f35d2f617bf9bd: Beginning new tablet copy session on tablet 02bddc1866e04858894e047ec100a6f0 from peer a4bf1df7fa6b4948be502c182bfbb833 at {username='slave'} at 127.30.32.70:40425: session id = a4bf1df7fa6b4948be502c182bfbb833-02bddc1866e04858894e047ec100a6f0
I20240502 20:46:07.061210 7690 ts_tablet_manager.cc:1398] T 28029d4e66d1453cbde9811ffc2a3283 P 201d2b6485084e7dad2cbb722af94779: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20240502 20:46:07.061556 4790 tablet_copy_source_session.cc:205] T 02bddc1866e04858894e047ec100a6f0 P ed47c023452a497192f35d2f617bf9bd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:07.062467 4528 tablet_copy_service.cc:342] P c3d26eafd51547acb780d4aac5ebe7cc: Request end of tablet copy session 201d2b6485084e7dad2cbb722af94779-28029d4e66d1453cbde9811ffc2a3283 received from {username='slave'} at 127.30.32.72:54159
I20240502 20:46:07.062549 4528 tablet_copy_service.cc:433] P c3d26eafd51547acb780d4aac5ebe7cc: ending tablet copy session 201d2b6485084e7dad2cbb722af94779-28029d4e66d1453cbde9811ffc2a3283 on tablet 28029d4e66d1453cbde9811ffc2a3283 with peer 201d2b6485084e7dad2cbb722af94779
I20240502 20:46:07.062665 7616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02bddc1866e04858894e047ec100a6f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:07.064934 7616 tablet_copy_client.cc:783] T 02bddc1866e04858894e047ec100a6f0 P a4bf1df7fa6b4948be502c182bfbb833: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:07.065111 7616 tablet_copy_client.cc:647] T 02bddc1866e04858894e047ec100a6f0 P a4bf1df7fa6b4948be502c182bfbb833: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:07.066426 7685 ts_tablet_manager.cc:902] T 428ffbac75db466191cb3bef63b4eb79 P a47e5c56f13843018ac584914a8d4707: Initiating tablet copy from peer 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:46:07.066912 7685 tablet_copy_client.cc:308] T 428ffbac75db466191cb3bef63b4eb79 P a47e5c56f13843018ac584914a8d4707: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.67:42141
I20240502 20:46:07.067406 4658 tablet_copy_service.cc:140] P 923cfab926e7443f8ffd7b21ad0a9d91: Received BeginTabletCopySession request for tablet 428ffbac75db466191cb3bef63b4eb79 from peer a47e5c56f13843018ac584914a8d4707 ({username='slave'} at 127.30.32.71:59213)
I20240502 20:46:07.067534 4658 tablet_copy_service.cc:161] P 923cfab926e7443f8ffd7b21ad0a9d91: Beginning new tablet copy session on tablet 428ffbac75db466191cb3bef63b4eb79 from peer a47e5c56f13843018ac584914a8d4707 at {username='slave'} at 127.30.32.71:59213: session id = a47e5c56f13843018ac584914a8d4707-428ffbac75db466191cb3bef63b4eb79
I20240502 20:46:07.067545 7616 tablet_copy_client.cc:523] T 02bddc1866e04858894e047ec100a6f0 P a4bf1df7fa6b4948be502c182bfbb833: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:07.068562 4658 tablet_copy_source_session.cc:205] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:07.068969 7616 tablet_bootstrap.cc:492] T 02bddc1866e04858894e047ec100a6f0 P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap starting.
I20240502 20:46:07.069440 7685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 428ffbac75db466191cb3bef63b4eb79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:07.071856 7685 tablet_copy_client.cc:783] T 428ffbac75db466191cb3bef63b4eb79 P a47e5c56f13843018ac584914a8d4707: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:07.072010 7685 tablet_copy_client.cc:647] T 428ffbac75db466191cb3bef63b4eb79 P a47e5c56f13843018ac584914a8d4707: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:07.073150 7685 tablet_copy_client.cc:523] T 428ffbac75db466191cb3bef63b4eb79 P a47e5c56f13843018ac584914a8d4707: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:07.074421 7685 tablet_bootstrap.cc:492] T 428ffbac75db466191cb3bef63b4eb79 P a47e5c56f13843018ac584914a8d4707: Bootstrap starting.
I20240502 20:46:07.093217 7803 ts_tablet_manager.cc:902] T f09dcfea3eb347e687f3c72f4ccc6929 P a47e5c56f13843018ac584914a8d4707: Initiating tablet copy from peer c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097)
I20240502 20:46:07.093909 7803 tablet_copy_client.cc:308] T f09dcfea3eb347e687f3c72f4ccc6929 P a47e5c56f13843018ac584914a8d4707: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.66:42097
I20240502 20:46:07.094353 4528 tablet_copy_service.cc:140] P c3d26eafd51547acb780d4aac5ebe7cc: Received BeginTabletCopySession request for tablet f09dcfea3eb347e687f3c72f4ccc6929 from peer a47e5c56f13843018ac584914a8d4707 ({username='slave'} at 127.30.32.71:46593)
I20240502 20:46:07.094456 4528 tablet_copy_service.cc:161] P c3d26eafd51547acb780d4aac5ebe7cc: Beginning new tablet copy session on tablet f09dcfea3eb347e687f3c72f4ccc6929 from peer a47e5c56f13843018ac584914a8d4707 at {username='slave'} at 127.30.32.71:46593: session id = a47e5c56f13843018ac584914a8d4707-f09dcfea3eb347e687f3c72f4ccc6929
I20240502 20:46:07.095453 4528 tablet_copy_source_session.cc:205] T f09dcfea3eb347e687f3c72f4ccc6929 P c3d26eafd51547acb780d4aac5ebe7cc: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:07.096261 7616 tablet_bootstrap.cc:492] T 02bddc1866e04858894e047ec100a6f0 P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:07.096568 7803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f09dcfea3eb347e687f3c72f4ccc6929. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:07.096580 7616 tablet_bootstrap.cc:492] T 02bddc1866e04858894e047ec100a6f0 P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap complete.
I20240502 20:46:07.096750 7616 ts_tablet_manager.cc:1367] T 02bddc1866e04858894e047ec100a6f0 P a4bf1df7fa6b4948be502c182bfbb833: Time spent bootstrapping tablet: real 0.028s user 0.021s sys 0.004s
I20240502 20:46:07.097273 7616 raft_consensus.cc:348] T 02bddc1866e04858894e047ec100a6f0 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } }
I20240502 20:46:07.097421 7616 raft_consensus.cc:729] T 02bddc1866e04858894e047ec100a6f0 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Initialized, Role: LEARNER
I20240502 20:46:07.097524 7616 consensus_queue.cc:260] T 02bddc1866e04858894e047ec100a6f0 P a4bf1df7fa6b4948be502c182bfbb833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } }
I20240502 20:46:07.097889 7616 ts_tablet_manager.cc:1398] T 02bddc1866e04858894e047ec100a6f0 P a4bf1df7fa6b4948be502c182bfbb833: Time spent starting tablet: real 0.001s user 0.005s sys 0.000s
I20240502 20:46:07.098476 4790 tablet_copy_service.cc:342] P ed47c023452a497192f35d2f617bf9bd: Request end of tablet copy session a4bf1df7fa6b4948be502c182bfbb833-02bddc1866e04858894e047ec100a6f0 received from {username='slave'} at 127.30.32.70:40425
I20240502 20:46:07.098552 4790 tablet_copy_service.cc:433] P ed47c023452a497192f35d2f617bf9bd: ending tablet copy session a4bf1df7fa6b4948be502c182bfbb833-02bddc1866e04858894e047ec100a6f0 on tablet 02bddc1866e04858894e047ec100a6f0 with peer a4bf1df7fa6b4948be502c182bfbb833
I20240502 20:46:07.099174 7685 tablet_bootstrap.cc:492] T 428ffbac75db466191cb3bef63b4eb79 P a47e5c56f13843018ac584914a8d4707: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:07.099429 7685 tablet_bootstrap.cc:492] T 428ffbac75db466191cb3bef63b4eb79 P a47e5c56f13843018ac584914a8d4707: Bootstrap complete.
I20240502 20:46:07.099587 7685 ts_tablet_manager.cc:1367] T 428ffbac75db466191cb3bef63b4eb79 P a47e5c56f13843018ac584914a8d4707: Time spent bootstrapping tablet: real 0.025s user 0.025s sys 0.000s
I20240502 20:46:07.100044 7685 raft_consensus.cc:348] T 428ffbac75db466191cb3bef63b4eb79 P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } }
I20240502 20:46:07.100170 7685 raft_consensus.cc:729] T 428ffbac75db466191cb3bef63b4eb79 P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Initialized, Role: LEARNER
I20240502 20:46:07.100333 7685 consensus_queue.cc:260] T 428ffbac75db466191cb3bef63b4eb79 P a47e5c56f13843018ac584914a8d4707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } }
I20240502 20:46:07.100531 7803 tablet_copy_client.cc:783] T f09dcfea3eb347e687f3c72f4ccc6929 P a47e5c56f13843018ac584914a8d4707: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:07.100620 7685 ts_tablet_manager.cc:1398] T 428ffbac75db466191cb3bef63b4eb79 P a47e5c56f13843018ac584914a8d4707: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:07.101119 4658 tablet_copy_service.cc:342] P 923cfab926e7443f8ffd7b21ad0a9d91: Request end of tablet copy session a47e5c56f13843018ac584914a8d4707-428ffbac75db466191cb3bef63b4eb79 received from {username='slave'} at 127.30.32.71:59213
I20240502 20:46:07.101200 4658 tablet_copy_service.cc:433] P 923cfab926e7443f8ffd7b21ad0a9d91: ending tablet copy session a47e5c56f13843018ac584914a8d4707-428ffbac75db466191cb3bef63b4eb79 on tablet 428ffbac75db466191cb3bef63b4eb79 with peer a47e5c56f13843018ac584914a8d4707
I20240502 20:46:07.101400 7803 tablet_copy_client.cc:647] T f09dcfea3eb347e687f3c72f4ccc6929 P a47e5c56f13843018ac584914a8d4707: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:07.101588 7616 ts_tablet_manager.cc:902] T 61f8c8d2925d46bdad8e903b48c3046e P a4bf1df7fa6b4948be502c182bfbb833: Initiating tablet copy from peer c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097)
I20240502 20:46:07.102294 7616 tablet_copy_client.cc:308] T 61f8c8d2925d46bdad8e903b48c3046e P a4bf1df7fa6b4948be502c182bfbb833: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.66:42097
I20240502 20:46:07.102718 4528 tablet_copy_service.cc:140] P c3d26eafd51547acb780d4aac5ebe7cc: Received BeginTabletCopySession request for tablet 61f8c8d2925d46bdad8e903b48c3046e from peer a4bf1df7fa6b4948be502c182bfbb833 ({username='slave'} at 127.30.32.70:46909)
I20240502 20:46:07.102829 7803 tablet_copy_client.cc:523] T f09dcfea3eb347e687f3c72f4ccc6929 P a47e5c56f13843018ac584914a8d4707: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:07.102844 4528 tablet_copy_service.cc:161] P c3d26eafd51547acb780d4aac5ebe7cc: Beginning new tablet copy session on tablet 61f8c8d2925d46bdad8e903b48c3046e from peer a4bf1df7fa6b4948be502c182bfbb833 at {username='slave'} at 127.30.32.70:46909: session id = a4bf1df7fa6b4948be502c182bfbb833-61f8c8d2925d46bdad8e903b48c3046e
I20240502 20:46:07.103677 4528 tablet_copy_source_session.cc:205] T 61f8c8d2925d46bdad8e903b48c3046e P c3d26eafd51547acb780d4aac5ebe7cc: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:07.103847 7803 tablet_bootstrap.cc:492] T f09dcfea3eb347e687f3c72f4ccc6929 P a47e5c56f13843018ac584914a8d4707: Bootstrap starting.
I20240502 20:46:07.105036 7690 ts_tablet_manager.cc:902] T 8e7e145762bf458a8374bfc503025a17 P 201d2b6485084e7dad2cbb722af94779: Initiating tablet copy from peer c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:46:07.105314 7616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61f8c8d2925d46bdad8e903b48c3046e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:07.105479 7690 tablet_copy_client.cc:308] T 8e7e145762bf458a8374bfc503025a17 P 201d2b6485084e7dad2cbb722af94779: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.65:33827
I20240502 20:46:07.107471 7616 tablet_copy_client.cc:783] T 61f8c8d2925d46bdad8e903b48c3046e P a4bf1df7fa6b4948be502c182bfbb833: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:07.107620 7616 tablet_copy_client.cc:647] T 61f8c8d2925d46bdad8e903b48c3046e P a4bf1df7fa6b4948be502c182bfbb833: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:07.109701 4397 tablet_copy_service.cc:140] P c92e6e46782849479c8a0c6d82276fb2: Received BeginTabletCopySession request for tablet 8e7e145762bf458a8374bfc503025a17 from peer 201d2b6485084e7dad2cbb722af94779 ({username='slave'} at 127.30.32.72:52725)
I20240502 20:46:07.109815 4397 tablet_copy_service.cc:161] P c92e6e46782849479c8a0c6d82276fb2: Beginning new tablet copy session on tablet 8e7e145762bf458a8374bfc503025a17 from peer 201d2b6485084e7dad2cbb722af94779 at {username='slave'} at 127.30.32.72:52725: session id = 201d2b6485084e7dad2cbb722af94779-8e7e145762bf458a8374bfc503025a17
I20240502 20:46:07.110428 7616 tablet_copy_client.cc:523] T 61f8c8d2925d46bdad8e903b48c3046e P a4bf1df7fa6b4948be502c182bfbb833: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:07.110765 4397 tablet_copy_source_session.cc:205] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:07.111613 7690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e7e145762bf458a8374bfc503025a17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:07.111681 7616 tablet_bootstrap.cc:492] T 61f8c8d2925d46bdad8e903b48c3046e P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap starting.
I20240502 20:46:07.113794 7690 tablet_copy_client.cc:783] T 8e7e145762bf458a8374bfc503025a17 P 201d2b6485084e7dad2cbb722af94779: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:07.113934 7690 tablet_copy_client.cc:647] T 8e7e145762bf458a8374bfc503025a17 P 201d2b6485084e7dad2cbb722af94779: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:07.115317 7690 tablet_copy_client.cc:523] T 8e7e145762bf458a8374bfc503025a17 P 201d2b6485084e7dad2cbb722af94779: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:07.116509 7690 tablet_bootstrap.cc:492] T 8e7e145762bf458a8374bfc503025a17 P 201d2b6485084e7dad2cbb722af94779: Bootstrap starting.
I20240502 20:46:07.124704 7803 tablet_bootstrap.cc:492] T f09dcfea3eb347e687f3c72f4ccc6929 P a47e5c56f13843018ac584914a8d4707: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:07.125131 7803 tablet_bootstrap.cc:492] T f09dcfea3eb347e687f3c72f4ccc6929 P a47e5c56f13843018ac584914a8d4707: Bootstrap complete.
I20240502 20:46:07.125356 7803 ts_tablet_manager.cc:1367] T f09dcfea3eb347e687f3c72f4ccc6929 P a47e5c56f13843018ac584914a8d4707: Time spent bootstrapping tablet: real 0.022s user 0.018s sys 0.004s
I20240502 20:46:07.125900 7803 raft_consensus.cc:348] T f09dcfea3eb347e687f3c72f4ccc6929 P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } }
I20240502 20:46:07.126098 7803 raft_consensus.cc:729] T f09dcfea3eb347e687f3c72f4ccc6929 P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Initialized, Role: LEARNER
I20240502 20:46:07.126258 7803 consensus_queue.cc:260] T f09dcfea3eb347e687f3c72f4ccc6929 P a47e5c56f13843018ac584914a8d4707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } }
I20240502 20:46:07.127367 7803 ts_tablet_manager.cc:1398] T f09dcfea3eb347e687f3c72f4ccc6929 P a47e5c56f13843018ac584914a8d4707: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:07.129366 4528 tablet_copy_service.cc:342] P c3d26eafd51547acb780d4aac5ebe7cc: Request end of tablet copy session a47e5c56f13843018ac584914a8d4707-f09dcfea3eb347e687f3c72f4ccc6929 received from {username='slave'} at 127.30.32.71:46593
I20240502 20:46:07.129436 4528 tablet_copy_service.cc:433] P c3d26eafd51547acb780d4aac5ebe7cc: ending tablet copy session a47e5c56f13843018ac584914a8d4707-f09dcfea3eb347e687f3c72f4ccc6929 on tablet f09dcfea3eb347e687f3c72f4ccc6929 with peer a47e5c56f13843018ac584914a8d4707
I20240502 20:46:07.132355 4235 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 20:46:07.136960 7616 tablet_bootstrap.cc:492] T 61f8c8d2925d46bdad8e903b48c3046e P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:07.137284 7616 tablet_bootstrap.cc:492] T 61f8c8d2925d46bdad8e903b48c3046e P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap complete.
I20240502 20:46:07.137475 7616 ts_tablet_manager.cc:1367] T 61f8c8d2925d46bdad8e903b48c3046e P a4bf1df7fa6b4948be502c182bfbb833: Time spent bootstrapping tablet: real 0.026s user 0.021s sys 0.004s
I20240502 20:46:07.137930 7616 raft_consensus.cc:348] T 61f8c8d2925d46bdad8e903b48c3046e P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } }
I20240502 20:46:07.138056 7616 raft_consensus.cc:729] T 61f8c8d2925d46bdad8e903b48c3046e P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Initialized, Role: LEARNER
I20240502 20:46:07.138161 7616 consensus_queue.cc:260] T 61f8c8d2925d46bdad8e903b48c3046e P a4bf1df7fa6b4948be502c182bfbb833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } }
I20240502 20:46:07.138456 7616 ts_tablet_manager.cc:1398] T 61f8c8d2925d46bdad8e903b48c3046e P a4bf1df7fa6b4948be502c182bfbb833: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:07.139215 7789 tablet.cc:1722] T 6f387855819c4af381e9e3d109904081 P a4bf1df7fa6b4948be502c182bfbb833: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 20:46:07.139534 7789 tablet.cc:1620] T 6f387855819c4af381e9e3d109904081 P a4bf1df7fa6b4948be502c182bfbb833: MemRowSet was empty: no flush needed.
I20240502 20:46:07.139590 7791 tablet.cc:1722] T 6f387855819c4af381e9e3d109904081 P ed47c023452a497192f35d2f617bf9bd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 20:46:07.139873 7791 tablet.cc:1620] T 6f387855819c4af381e9e3d109904081 P ed47c023452a497192f35d2f617bf9bd: MemRowSet was empty: no flush needed.
I20240502 20:46:07.142105 4528 tablet_copy_service.cc:342] P c3d26eafd51547acb780d4aac5ebe7cc: Request end of tablet copy session a4bf1df7fa6b4948be502c182bfbb833-61f8c8d2925d46bdad8e903b48c3046e received from {username='slave'} at 127.30.32.70:46909
I20240502 20:46:07.142195 4528 tablet_copy_service.cc:433] P c3d26eafd51547acb780d4aac5ebe7cc: ending tablet copy session a4bf1df7fa6b4948be502c182bfbb833-61f8c8d2925d46bdad8e903b48c3046e on tablet 61f8c8d2925d46bdad8e903b48c3046e with peer a4bf1df7fa6b4948be502c182bfbb833
I20240502 20:46:07.142920 6831 raft_consensus.cc:1203] T d37730b109d04b6395f75c7552524877 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:07.143970 7790 tablet.cc:1722] T 6f387855819c4af381e9e3d109904081 P 7845c397230d49358ba2fad9db53f97c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 20:46:07.144256 7790 tablet.cc:1620] T 6f387855819c4af381e9e3d109904081 P 7845c397230d49358ba2fad9db53f97c: MemRowSet was empty: no flush needed.
I20240502 20:46:07.145607 7690 tablet_bootstrap.cc:492] T 8e7e145762bf458a8374bfc503025a17 P 201d2b6485084e7dad2cbb722af94779: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:07.145879 7690 tablet_bootstrap.cc:492] T 8e7e145762bf458a8374bfc503025a17 P 201d2b6485084e7dad2cbb722af94779: Bootstrap complete.
I20240502 20:46:07.146039 7690 ts_tablet_manager.cc:1367] T 8e7e145762bf458a8374bfc503025a17 P 201d2b6485084e7dad2cbb722af94779: Time spent bootstrapping tablet: real 0.030s user 0.024s sys 0.003s
I20240502 20:46:07.146530 7690 raft_consensus.cc:348] T 8e7e145762bf458a8374bfc503025a17 P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } }
I20240502 20:46:07.146646 7690 raft_consensus.cc:729] T 8e7e145762bf458a8374bfc503025a17 P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Initialized, Role: LEARNER
I20240502 20:46:07.146756 7690 consensus_queue.cc:260] T 8e7e145762bf458a8374bfc503025a17 P 201d2b6485084e7dad2cbb722af94779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } }
I20240502 20:46:07.147040 7690 ts_tablet_manager.cc:1398] T 8e7e145762bf458a8374bfc503025a17 P 201d2b6485084e7dad2cbb722af94779: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:07.147795 4397 tablet_copy_service.cc:342] P c92e6e46782849479c8a0c6d82276fb2: Request end of tablet copy session 201d2b6485084e7dad2cbb722af94779-8e7e145762bf458a8374bfc503025a17 received from {username='slave'} at 127.30.32.72:52725
I20240502 20:46:07.147897 4397 tablet_copy_service.cc:433] P c92e6e46782849479c8a0c6d82276fb2: ending tablet copy session 201d2b6485084e7dad2cbb722af94779-8e7e145762bf458a8374bfc503025a17 on tablet 8e7e145762bf458a8374bfc503025a17 with peer 201d2b6485084e7dad2cbb722af94779
I20240502 20:46:07.148316 4230 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: rebalancer_dynamic_table [id=e0660e829abf435592bc23d271955609] alter complete (version 2)
I20240502 20:46:07.152232 7695 ts_tablet_manager.cc:902] T d37fab152a8a44868d3160ec3057bb98 P 7845c397230d49358ba2fad9db53f97c: Initiating tablet copy from peer ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:46:07.152834 7695 tablet_copy_client.cc:308] T d37fab152a8a44868d3160ec3057bb98 P 7845c397230d49358ba2fad9db53f97c: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.68:38193
I20240502 20:46:07.153702 4790 tablet_copy_service.cc:140] P ed47c023452a497192f35d2f617bf9bd: Received BeginTabletCopySession request for tablet d37fab152a8a44868d3160ec3057bb98 from peer 7845c397230d49358ba2fad9db53f97c ({username='slave'} at 127.30.32.69:43573)
I20240502 20:46:07.153829 4790 tablet_copy_service.cc:161] P ed47c023452a497192f35d2f617bf9bd: Beginning new tablet copy session on tablet d37fab152a8a44868d3160ec3057bb98 from peer 7845c397230d49358ba2fad9db53f97c at {username='slave'} at 127.30.32.69:43573: session id = 7845c397230d49358ba2fad9db53f97c-d37fab152a8a44868d3160ec3057bb98
I20240502 20:46:07.154866 4790 tablet_copy_source_session.cc:205] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:07.155843 7695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d37fab152a8a44868d3160ec3057bb98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:07.158043 7695 tablet_copy_client.cc:783] T d37fab152a8a44868d3160ec3057bb98 P 7845c397230d49358ba2fad9db53f97c: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:07.158202 7695 tablet_copy_client.cc:647] T d37fab152a8a44868d3160ec3057bb98 P 7845c397230d49358ba2fad9db53f97c: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:07.159436 7695 tablet_copy_client.cc:523] T d37fab152a8a44868d3160ec3057bb98 P 7845c397230d49358ba2fad9db53f97c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:07.160753 7695 tablet_bootstrap.cc:492] T d37fab152a8a44868d3160ec3057bb98 P 7845c397230d49358ba2fad9db53f97c: Bootstrap starting.
I20240502 20:46:07.184528 7695 tablet_bootstrap.cc:492] T d37fab152a8a44868d3160ec3057bb98 P 7845c397230d49358ba2fad9db53f97c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:07.184929 7695 tablet_bootstrap.cc:492] T d37fab152a8a44868d3160ec3057bb98 P 7845c397230d49358ba2fad9db53f97c: Bootstrap complete.
I20240502 20:46:07.185329 7695 ts_tablet_manager.cc:1367] T d37fab152a8a44868d3160ec3057bb98 P 7845c397230d49358ba2fad9db53f97c: Time spent bootstrapping tablet: real 0.025s user 0.025s sys 0.000s
I20240502 20:46:07.185803 7695 raft_consensus.cc:348] T d37fab152a8a44868d3160ec3057bb98 P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } }
I20240502 20:46:07.185933 7695 raft_consensus.cc:729] T d37fab152a8a44868d3160ec3057bb98 P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Initialized, Role: LEARNER
I20240502 20:46:07.186046 7695 consensus_queue.cc:260] T d37fab152a8a44868d3160ec3057bb98 P 7845c397230d49358ba2fad9db53f97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } }
I20240502 20:46:07.186373 7695 ts_tablet_manager.cc:1398] T d37fab152a8a44868d3160ec3057bb98 P 7845c397230d49358ba2fad9db53f97c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:07.186949 4790 tablet_copy_service.cc:342] P ed47c023452a497192f35d2f617bf9bd: Request end of tablet copy session 7845c397230d49358ba2fad9db53f97c-d37fab152a8a44868d3160ec3057bb98 received from {username='slave'} at 127.30.32.69:43573
I20240502 20:46:07.187037 4790 tablet_copy_service.cc:433] P ed47c023452a497192f35d2f617bf9bd: ending tablet copy session 7845c397230d49358ba2fad9db53f97c-d37fab152a8a44868d3160ec3057bb98 on tablet d37fab152a8a44868d3160ec3057bb98 with peer 7845c397230d49358ba2fad9db53f97c
I20240502 20:46:07.215277 7310 consensus_queue.cc:1035] T 935f0efc1fe146f4ac21fac4c0cc688c P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:07.221320 7322 consensus_queue.cc:1035] T 935f0efc1fe146f4ac21fac4c0cc688c P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:07.237955 7098 raft_consensus.cc:1203] T f352efa74d3c4da3907efdc37237211f P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:07.250710 4233 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 20:46:07.254837 6702 raft_consensus.cc:1203] T a2e81d2c63cd4e4cac5086506b5c4a12 P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:07.256453 6831 raft_consensus.cc:1203] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:07.259502 7789 tablet.cc:1722] T 6f387855819c4af381e9e3d109904081 P a4bf1df7fa6b4948be502c182bfbb833: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 20:46:07.259915 7790 tablet.cc:1722] T 6f387855819c4af381e9e3d109904081 P 7845c397230d49358ba2fad9db53f97c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 20:46:07.262341 7168 consensus_queue.cc:1035] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:07.263186 7168 consensus_queue.cc:1035] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:07.266315 7791 tablet.cc:1722] T 6f387855819c4af381e9e3d109904081 P ed47c023452a497192f35d2f617bf9bd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 20:46:07.268656 6957 raft_consensus.cc:1203] T f6d4a3019ba54eb8b3506b88f04a6262 P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:07.273589 4235 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: rebalancer_dynamic_table_new_name [id=e0660e829abf435592bc23d271955609] alter complete (version 3)
I20240502 20:46:07.277385 4230 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240502 20:46:07.277521 4230 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240502 20:46:07.280603 4230 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: Sending DeleteTablet for 4 replicas of tablet 61a6d9a8e21e4a098cfad7281f0e0fd6
I20240502 20:46:07.281098 4488 tablet_service.cc:1514] Processing DeleteTablet for tablet 61a6d9a8e21e4a098cfad7281f0e0fd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:07 UTC) from {username='slave'} at 127.0.0.1:48890
I20240502 20:46:07.281244 4749 tablet_service.cc:1514] Processing DeleteTablet for tablet 61a6d9a8e21e4a098cfad7281f0e0fd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:07 UTC) from {username='slave'} at 127.0.0.1:38526
I20240502 20:46:07.281288 4356 tablet_service.cc:1514] Processing DeleteTablet for tablet 61a6d9a8e21e4a098cfad7281f0e0fd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:07 UTC) from {username='slave'} at 127.0.0.1:38306
I20240502 20:46:07.281401 6809 tablet_service.cc:1514] Processing DeleteTablet for tablet 61a6d9a8e21e4a098cfad7281f0e0fd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:07 UTC) from {username='slave'} at 127.0.0.1:35290
I20240502 20:46:07.281477 7299 tablet_replica.cc:330] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P ed47c023452a497192f35d2f617bf9bd: stopping tablet replica
I20240502 20:46:07.281530 7297 tablet_replica.cc:330] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c92e6e46782849479c8a0c6d82276fb2: stopping tablet replica
I20240502 20:46:07.281616 7299 raft_consensus.cc:2229] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:07.281656 7297 raft_consensus.cc:2229] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:07.281666 7908 tablet_replica.cc:330] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P a4bf1df7fa6b4948be502c182bfbb833: stopping tablet replica
I20240502 20:46:07.281797 7299 raft_consensus.cc:2258] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:07.281807 7297 raft_consensus.cc:2258] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:07.281793 7908 raft_consensus.cc:2229] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEARNER]: Raft consensus shutting down.
I20240502 20:46:07.281904 7908 raft_consensus.cc:2258] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEARNER]: Raft consensus is shut down!
I20240502 20:46:07.282505 7908 ts_tablet_manager.cc:1875] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P a4bf1df7fa6b4948be502c182bfbb833: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:07.282812 7297 ts_tablet_manager.cc:1875] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c92e6e46782849479c8a0c6d82276fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:07.282800 7299 ts_tablet_manager.cc:1875] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P ed47c023452a497192f35d2f617bf9bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:07.283504 7907 tablet_replica.cc:330] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c3d26eafd51547acb780d4aac5ebe7cc: stopping tablet replica
I20240502 20:46:07.283638 7907 raft_consensus.cc:2229] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:07.283954 7907 raft_consensus.cc:2258] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:07.284765 7908 ts_tablet_manager.cc:1888] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P a4bf1df7fa6b4948be502c182bfbb833: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:07.284828 7908 log.cc:1196] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P a4bf1df7fa6b4948be502c182bfbb833: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/wal/wals/61a6d9a8e21e4a098cfad7281f0e0fd6
I20240502 20:46:07.284893 7299 ts_tablet_manager.cc:1888] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P ed47c023452a497192f35d2f617bf9bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:07.284893 7297 ts_tablet_manager.cc:1888] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c92e6e46782849479c8a0c6d82276fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:07.285135 7908 ts_tablet_manager.cc:1909] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P a4bf1df7fa6b4948be502c182bfbb833: Deleting consensus metadata
I20240502 20:46:07.284960 7297 log.cc:1196] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c92e6e46782849479c8a0c6d82276fb2: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal/wals/61a6d9a8e21e4a098cfad7281f0e0fd6
I20240502 20:46:07.284960 7299 log.cc:1196] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P ed47c023452a497192f35d2f617bf9bd: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal/wals/61a6d9a8e21e4a098cfad7281f0e0fd6
I20240502 20:46:07.285527 7297 ts_tablet_manager.cc:1909] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c92e6e46782849479c8a0c6d82276fb2: Deleting consensus metadata
I20240502 20:46:07.285670 7299 ts_tablet_manager.cc:1909] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P ed47c023452a497192f35d2f617bf9bd: Deleting consensus metadata
I20240502 20:46:07.285912 7907 ts_tablet_manager.cc:1875] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:07.281023 4230 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: Sending DeleteTablet for 3 replicas of tablet 3ecf52a5823142beaf4217419be1d01c
I20240502 20:46:07.285861 4220 catalog_manager.cc:4830] TS a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867): tablet 61a6d9a8e21e4a098cfad7281f0e0fd6 (table rebalance_test_table_6 [id=f4c78cd17a604c19aba2b334e324fc04]) successfully deleted
I20240502 20:46:07.286314 4222 catalog_manager.cc:4830] TS ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193): tablet 61a6d9a8e21e4a098cfad7281f0e0fd6 (table rebalance_test_table_6 [id=f4c78cd17a604c19aba2b334e324fc04]) successfully deleted
I20240502 20:46:07.286556 4222 catalog_manager.cc:4830] TS c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827): tablet 61a6d9a8e21e4a098cfad7281f0e0fd6 (table rebalance_test_table_6 [id=f4c78cd17a604c19aba2b334e324fc04]) successfully deleted
I20240502 20:46:07.286939 4356 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ecf52a5823142beaf4217419be1d01c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:07 UTC) from {username='slave'} at 127.0.0.1:38306
I20240502 20:46:07.286942 4618 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ecf52a5823142beaf4217419be1d01c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:07 UTC) from {username='slave'} at 127.0.0.1:57068
I20240502 20:46:07.287124 7297 tablet_replica.cc:330] T 3ecf52a5823142beaf4217419be1d01c P c92e6e46782849479c8a0c6d82276fb2: stopping tablet replica
I20240502 20:46:07.287266 7297 raft_consensus.cc:2229] T 3ecf52a5823142beaf4217419be1d01c P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:07.287317 4749 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ecf52a5823142beaf4217419be1d01c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:07 UTC) from {username='slave'} at 127.0.0.1:38526
I20240502 20:46:07.287417 7297 raft_consensus.cc:2258] T 3ecf52a5823142beaf4217419be1d01c P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:07.287503 7299 tablet_replica.cc:330] T 3ecf52a5823142beaf4217419be1d01c P ed47c023452a497192f35d2f617bf9bd: stopping tablet replica
I20240502 20:46:07.287632 7299 raft_consensus.cc:2229] T 3ecf52a5823142beaf4217419be1d01c P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:07.287787 7299 raft_consensus.cc:2258] T 3ecf52a5823142beaf4217419be1d01c P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:07.287942 7297 ts_tablet_manager.cc:1875] T 3ecf52a5823142beaf4217419be1d01c P c92e6e46782849479c8a0c6d82276fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:07.288161 7299 ts_tablet_manager.cc:1875] T 3ecf52a5823142beaf4217419be1d01c P ed47c023452a497192f35d2f617bf9bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:07.288607 7907 ts_tablet_manager.cc:1888] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c3d26eafd51547acb780d4aac5ebe7cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:07.288669 7907 log.cc:1196] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal/wals/61a6d9a8e21e4a098cfad7281f0e0fd6
I20240502 20:46:07.288973 7907 ts_tablet_manager.cc:1909] T 61a6d9a8e21e4a098cfad7281f0e0fd6 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting consensus metadata
I20240502 20:46:07.289085 4230 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: Sending DeleteTablet for 4 replicas of tablet f6d4a3019ba54eb8b3506b88f04a6262
I20240502 20:46:07.289691 7913 tablet_replica.cc:330] T 3ecf52a5823142beaf4217419be1d01c P 923cfab926e7443f8ffd7b21ad0a9d91: stopping tablet replica
I20240502 20:46:07.289716 4221 catalog_manager.cc:4830] TS c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097): tablet 61a6d9a8e21e4a098cfad7281f0e0fd6 (table rebalance_test_table_6 [id=f4c78cd17a604c19aba2b334e324fc04]) successfully deleted
I20240502 20:46:07.289752 4488 tablet_service.cc:1514] Processing DeleteTablet for tablet f6d4a3019ba54eb8b3506b88f04a6262 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:07 UTC) from {username='slave'} at 127.0.0.1:48890
I20240502 20:46:07.289888 7907 tablet_replica.cc:330] T f6d4a3019ba54eb8b3506b88f04a6262 P c3d26eafd51547acb780d4aac5ebe7cc: stopping tablet replica
I20240502 20:46:07.289876 7913 raft_consensus.cc:2229] T 3ecf52a5823142beaf4217419be1d01c P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:07.289896 4749 tablet_service.cc:1514] Processing DeleteTablet for tablet f6d4a3019ba54eb8b3506b88f04a6262 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:07 UTC) from {username='slave'} at 127.0.0.1:38526
I20240502 20:46:07.289950 7297 ts_tablet_manager.cc:1888] T 3ecf52a5823142beaf4217419be1d01c P c92e6e46782849479c8a0c6d82276fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 20:46:07.290005 7297 log.cc:1196] T 3ecf52a5823142beaf4217419be1d01c P c92e6e46782849479c8a0c6d82276fb2: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal/wals/3ecf52a5823142beaf4217419be1d01c
I20240502 20:46:07.290024 7913 raft_consensus.cc:2258] T 3ecf52a5823142beaf4217419be1d01c P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:07.290045 7907 raft_consensus.cc:2229] T f6d4a3019ba54eb8b3506b88f04a6262 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:07.290232 7907 raft_consensus.cc:2258] T f6d4a3019ba54eb8b3506b88f04a6262 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:07.290282 7297 ts_tablet_manager.cc:1909] T 3ecf52a5823142beaf4217419be1d01c P c92e6e46782849479c8a0c6d82276fb2: Deleting consensus metadata
I20240502 20:46:07.290575 7913 ts_tablet_manager.cc:1875] T 3ecf52a5823142beaf4217419be1d01c P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:07.290567 6946 tablet_service.cc:1514] Processing DeleteTablet for tablet f6d4a3019ba54eb8b3506b88f04a6262 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:07 UTC) from {username='slave'} at 127.0.0.1:45156
I20240502 20:46:07.290849 4222 catalog_manager.cc:4830] TS c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827): tablet 3ecf52a5823142beaf4217419be1d01c (table rebalance_test_table_6 [id=f4c78cd17a604c19aba2b334e324fc04]) successfully deleted
I20240502 20:46:07.291121 4356 tablet_service.cc:1514] Processing DeleteTablet for tablet f6d4a3019ba54eb8b3506b88f04a6262 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:07 UTC) from {username='slave'} at 127.0.0.1:38306
I20240502 20:46:07.291577 7297 tablet_replica.cc:330] T f6d4a3019ba54eb8b3506b88f04a6262 P c92e6e46782849479c8a0c6d82276fb2: stopping tablet replica
I20240502 20:46:07.291765 7297 raft_consensus.cc:2229] T f6d4a3019ba54eb8b3506b88f04a6262 P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:07.291812 7299 ts_tablet_manager.cc:1888] T 3ecf52a5823142beaf4217419be1d01c P ed47c023452a497192f35d2f617bf9bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 20:46:07.291922 7907 ts_tablet_manager.cc:1875] T f6d4a3019ba54eb8b3506b88f04a6262 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:07.292096 7297 raft_consensus.cc:2258] T f6d4a3019ba54eb8b3506b88f04a6262 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:07.292753 7297 ts_tablet_manager.cc:1875] T f6d4a3019ba54eb8b3506b88f04a6262 P c92e6e46782849479c8a0c6d82276fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:07.293057 7299 log.cc:1196] T 3ecf52a5823142beaf4217419be1d01c P ed47c023452a497192f35d2f617bf9bd: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal/wals/3ecf52a5823142beaf4217419be1d01c
I20240502 20:46:07.293404 7299 ts_tablet_manager.cc:1909] T 3ecf52a5823142beaf4217419be1d01c P ed47c023452a497192f35d2f617bf9bd: Deleting consensus metadata
I20240502 20:46:07.293666 7913 ts_tablet_manager.cc:1888] T 3ecf52a5823142beaf4217419be1d01c P 923cfab926e7443f8ffd7b21ad0a9d91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 20:46:07.293797 7907 ts_tablet_manager.cc:1888] T f6d4a3019ba54eb8b3506b88f04a6262 P c3d26eafd51547acb780d4aac5ebe7cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:07.293850 7913 log.cc:1196] T 3ecf52a5823142beaf4217419be1d01c P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal/wals/3ecf52a5823142beaf4217419be1d01c
I20240502 20:46:07.294021 7299 tablet_replica.cc:330] T f6d4a3019ba54eb8b3506b88f04a6262 P ed47c023452a497192f35d2f617bf9bd: stopping tablet replica
I20240502 20:46:07.294109 4222 catalog_manager.cc:4830] TS ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193): tablet 3ecf52a5823142beaf4217419be1d01c (table rebalance_test_table_6 [id=f4c78cd17a604c19aba2b334e324fc04]) successfully deleted
I20240502 20:46:07.294152 7299 raft_consensus.cc:2229] T f6d4a3019ba54eb8b3506b88f04a6262 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:07.293929 7907 log.cc:1196] T f6d4a3019ba54eb8b3506b88f04a6262 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal/wals/f6d4a3019ba54eb8b3506b88f04a6262
I20240502 20:46:07.294302 7299 raft_consensus.cc:2258] T f6d4a3019ba54eb8b3506b88f04a6262 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:07.294672 7297 ts_tablet_manager.cc:1888] T f6d4a3019ba54eb8b3506b88f04a6262 P c92e6e46782849479c8a0c6d82276fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:07.294749 7913 ts_tablet_manager.cc:1909] T 3ecf52a5823142beaf4217419be1d01c P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting consensus metadata
I20240502 20:46:07.294888 7297 log.cc:1196] T f6d4a3019ba54eb8b3506b88f04a6262 P c92e6e46782849479c8a0c6d82276fb2: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal/wals/f6d4a3019ba54eb8b3506b88f04a6262
I20240502 20:46:07.294952 7907 ts_tablet_manager.cc:1909] T f6d4a3019ba54eb8b3506b88f04a6262 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting consensus metadata
I20240502 20:46:07.295037 7299 ts_tablet_manager.cc:1875] T f6d4a3019ba54eb8b3506b88f04a6262 P ed47c023452a497192f35d2f617bf9bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:07.295203 7297 ts_tablet_manager.cc:1909] T f6d4a3019ba54eb8b3506b88f04a6262 P c92e6e46782849479c8a0c6d82276fb2: Deleting consensus metadata
I20240502 20:46:07.295473 4219 catalog_manager.cc:4830] TS 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141): tablet 3ecf52a5823142beaf4217419be1d01c (table rebalance_test_table_6 [id=f4c78cd17a604c19aba2b334e324fc04]) successfully deleted
I20240502 20:46:07.296653 4222 catalog_manager.cc:4830] TS c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827): tablet f6d4a3019ba54eb8b3506b88f04a6262 (table rebalance_test_table_6 [id=f4c78cd17a604c19aba2b334e324fc04]) successfully deleted
I20240502 20:46:07.297040 4221 catalog_manager.cc:4830] TS c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097): tablet f6d4a3019ba54eb8b3506b88f04a6262 (table rebalance_test_table_6 [id=f4c78cd17a604c19aba2b334e324fc04]) successfully deleted
I20240502 20:46:07.297080 7299 ts_tablet_manager.cc:1888] T f6d4a3019ba54eb8b3506b88f04a6262 P ed47c023452a497192f35d2f617bf9bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:07.297142 7299 log.cc:1196] T f6d4a3019ba54eb8b3506b88f04a6262 P ed47c023452a497192f35d2f617bf9bd: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal/wals/f6d4a3019ba54eb8b3506b88f04a6262
I20240502 20:46:07.297227 7915 tablet_replica.cc:330] T f6d4a3019ba54eb8b3506b88f04a6262 P a47e5c56f13843018ac584914a8d4707: stopping tablet replica
I20240502 20:46:07.297448 7915 raft_consensus.cc:2229] T f6d4a3019ba54eb8b3506b88f04a6262 P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Raft consensus shutting down.
I20240502 20:46:07.297580 7915 raft_consensus.cc:2258] T f6d4a3019ba54eb8b3506b88f04a6262 P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Raft consensus is shut down!
I20240502 20:46:07.297937 7299 ts_tablet_manager.cc:1909] T f6d4a3019ba54eb8b3506b88f04a6262 P ed47c023452a497192f35d2f617bf9bd: Deleting consensus metadata
I20240502 20:46:07.298398 7915 ts_tablet_manager.cc:1875] T f6d4a3019ba54eb8b3506b88f04a6262 P a47e5c56f13843018ac584914a8d4707: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:07.298699 4222 catalog_manager.cc:4830] TS ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193): tablet f6d4a3019ba54eb8b3506b88f04a6262 (table rebalance_test_table_6 [id=f4c78cd17a604c19aba2b334e324fc04]) successfully deleted
I20240502 20:46:07.300662 7915 ts_tablet_manager.cc:1888] T f6d4a3019ba54eb8b3506b88f04a6262 P a47e5c56f13843018ac584914a8d4707: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:07.300801 7915 log.cc:1196] T f6d4a3019ba54eb8b3506b88f04a6262 P a47e5c56f13843018ac584914a8d4707: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/wal/wals/f6d4a3019ba54eb8b3506b88f04a6262
I20240502 20:46:07.301198 7915 ts_tablet_manager.cc:1909] T f6d4a3019ba54eb8b3506b88f04a6262 P a47e5c56f13843018ac584914a8d4707: Deleting consensus metadata
I20240502 20:46:07.301776 4222 catalog_manager.cc:4830] TS a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): tablet f6d4a3019ba54eb8b3506b88f04a6262 (table rebalance_test_table_6 [id=f4c78cd17a604c19aba2b334e324fc04]) successfully deleted
I20240502 20:46:07.303097 7098 raft_consensus.cc:1203] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:07.339426 6957 raft_consensus.cc:1203] T 47c5087093214fbeae35590b4b3c3d42 P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:07.377243 6702 raft_consensus.cc:1203] T a62a7ce526f14c25994bb9e0e3c3bc84 P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:07.377779 7098 raft_consensus.cc:1203] T 9e34e8e17a7e413b97eb614a7df727d5 P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:07.377768 4230 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 20:46:07.377923 4230 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 20:46:07.381162 4230 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: Sending DeleteTablet for 3 replicas of tablet 6f387855819c4af381e9e3d109904081
I20240502 20:46:07.381759 6809 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f387855819c4af381e9e3d109904081 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:07 UTC) from {username='slave'} at 127.0.0.1:35290
I20240502 20:46:07.381768 4749 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f387855819c4af381e9e3d109904081 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:07 UTC) from {username='slave'} at 127.0.0.1:38526
I20240502 20:46:07.381951 7299 tablet_replica.cc:330] T 6f387855819c4af381e9e3d109904081 P ed47c023452a497192f35d2f617bf9bd: stopping tablet replica
I20240502 20:46:07.381960 7908 tablet_replica.cc:330] T 6f387855819c4af381e9e3d109904081 P a4bf1df7fa6b4948be502c182bfbb833: stopping tablet replica
I20240502 20:46:07.382091 7908 raft_consensus.cc:2229] T 6f387855819c4af381e9e3d109904081 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:07.382103 7299 raft_consensus.cc:2229] T 6f387855819c4af381e9e3d109904081 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:07.382223 7299 raft_consensus.cc:2258] T 6f387855819c4af381e9e3d109904081 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:07.382303 7908 raft_consensus.cc:2258] T 6f387855819c4af381e9e3d109904081 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:07.382673 6683 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f387855819c4af381e9e3d109904081 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:07 UTC) from {username='slave'} at 127.0.0.1:37506
I20240502 20:46:07.382923 7299 ts_tablet_manager.cc:1875] T 6f387855819c4af381e9e3d109904081 P ed47c023452a497192f35d2f617bf9bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:07.382957 7908 ts_tablet_manager.cc:1875] T 6f387855819c4af381e9e3d109904081 P a4bf1df7fa6b4948be502c182bfbb833: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:07.382987 7938 tablet_replica.cc:330] T 6f387855819c4af381e9e3d109904081 P 7845c397230d49358ba2fad9db53f97c: stopping tablet replica
I20240502 20:46:07.383100 7938 raft_consensus.cc:2229] T 6f387855819c4af381e9e3d109904081 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:07.383251 7938 raft_consensus.cc:2258] T 6f387855819c4af381e9e3d109904081 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:07.383975 7938 ts_tablet_manager.cc:1875] T 6f387855819c4af381e9e3d109904081 P 7845c397230d49358ba2fad9db53f97c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:07.385069 7299 ts_tablet_manager.cc:1888] T 6f387855819c4af381e9e3d109904081 P ed47c023452a497192f35d2f617bf9bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:07.385123 7908 ts_tablet_manager.cc:1888] T 6f387855819c4af381e9e3d109904081 P a4bf1df7fa6b4948be502c182bfbb833: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:07.385162 7299 log.cc:1196] T 6f387855819c4af381e9e3d109904081 P ed47c023452a497192f35d2f617bf9bd: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal/wals/6f387855819c4af381e9e3d109904081
I20240502 20:46:07.385427 7299 ts_tablet_manager.cc:1909] T 6f387855819c4af381e9e3d109904081 P ed47c023452a497192f35d2f617bf9bd: Deleting consensus metadata
I20240502 20:46:07.385236 7908 log.cc:1196] T 6f387855819c4af381e9e3d109904081 P a4bf1df7fa6b4948be502c182bfbb833: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/wal/wals/6f387855819c4af381e9e3d109904081
I20240502 20:46:07.386045 7908 ts_tablet_manager.cc:1909] T 6f387855819c4af381e9e3d109904081 P a4bf1df7fa6b4948be502c182bfbb833: Deleting consensus metadata
I20240502 20:46:07.386919 7938 ts_tablet_manager.cc:1888] T 6f387855819c4af381e9e3d109904081 P 7845c397230d49358ba2fad9db53f97c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:07.387063 4222 catalog_manager.cc:4830] TS ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193): tablet 6f387855819c4af381e9e3d109904081 (table rebalancer_dynamic_table_new_name [id=e0660e829abf435592bc23d271955609]) successfully deleted
I20240502 20:46:07.387255 7938 log.cc:1196] T 6f387855819c4af381e9e3d109904081 P 7845c397230d49358ba2fad9db53f97c: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/wal/wals/6f387855819c4af381e9e3d109904081
I20240502 20:46:07.387531 7938 ts_tablet_manager.cc:1909] T 6f387855819c4af381e9e3d109904081 P 7845c397230d49358ba2fad9db53f97c: Deleting consensus metadata
I20240502 20:46:07.387569 4220 catalog_manager.cc:4830] TS a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867): tablet 6f387855819c4af381e9e3d109904081 (table rebalancer_dynamic_table_new_name [id=e0660e829abf435592bc23d271955609]) successfully deleted
I20240502 20:46:07.388146 4221 catalog_manager.cc:4830] TS 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): tablet 6f387855819c4af381e9e3d109904081 (table rebalancer_dynamic_table_new_name [id=e0660e829abf435592bc23d271955609]) successfully deleted
I20240502 20:46:07.397409 6831 raft_consensus.cc:1203] T 7abfd61f55554042978b50761cbd6ab5 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:07.414088 6702 raft_consensus.cc:1203] T d6fb394fdcff436886fe0634d79c024c P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:07.418231 7098 raft_consensus.cc:1203] T 28029d4e66d1453cbde9811ffc2a3283 P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:07.482952 4230 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41812:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 20:46:07.488381 4356 tablet_service.cc:1467] Processing CreateTablet for tablet a25f2855fb7d4eac9f1c920fdd01e2ba (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0d23318bd0f5482face4d225e00e7757]), partition=
I20240502 20:46:07.488786 4356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a25f2855fb7d4eac9f1c920fdd01e2ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:07.488811 6946 tablet_service.cc:1467] Processing CreateTablet for tablet a25f2855fb7d4eac9f1c920fdd01e2ba (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0d23318bd0f5482face4d225e00e7757]), partition=
I20240502 20:46:07.489089 6946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a25f2855fb7d4eac9f1c920fdd01e2ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:07.490898 7939 tablet_bootstrap.cc:492] T a25f2855fb7d4eac9f1c920fdd01e2ba P a47e5c56f13843018ac584914a8d4707: Bootstrap starting.
I20240502 20:46:07.490906 7940 tablet_bootstrap.cc:492] T a25f2855fb7d4eac9f1c920fdd01e2ba P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:46:07.491267 7078 tablet_service.cc:1467] Processing CreateTablet for tablet a25f2855fb7d4eac9f1c920fdd01e2ba (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0d23318bd0f5482face4d225e00e7757]), partition=
I20240502 20:46:07.491518 7078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a25f2855fb7d4eac9f1c920fdd01e2ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:07.491819 7939 tablet_bootstrap.cc:654] T a25f2855fb7d4eac9f1c920fdd01e2ba P a47e5c56f13843018ac584914a8d4707: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:07.491887 7940 tablet_bootstrap.cc:654] T a25f2855fb7d4eac9f1c920fdd01e2ba P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:07.492776 7939 tablet_bootstrap.cc:492] T a25f2855fb7d4eac9f1c920fdd01e2ba P a47e5c56f13843018ac584914a8d4707: No bootstrap required, opened a new log
I20240502 20:46:07.492838 7939 ts_tablet_manager.cc:1367] T a25f2855fb7d4eac9f1c920fdd01e2ba P a47e5c56f13843018ac584914a8d4707: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:07.493398 7939 raft_consensus.cc:348] T a25f2855fb7d4eac9f1c920fdd01e2ba P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:07.493930 7943 tablet_bootstrap.cc:492] T a25f2855fb7d4eac9f1c920fdd01e2ba P 201d2b6485084e7dad2cbb722af94779: Bootstrap starting.
I20240502 20:46:07.493983 7939 raft_consensus.cc:374] T a25f2855fb7d4eac9f1c920fdd01e2ba P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:07.494040 7939 raft_consensus.cc:729] T a25f2855fb7d4eac9f1c920fdd01e2ba P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Initialized, Role: FOLLOWER
I20240502 20:46:07.494205 7940 tablet_bootstrap.cc:492] T a25f2855fb7d4eac9f1c920fdd01e2ba P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:46:07.494207 7939 consensus_queue.cc:260] T a25f2855fb7d4eac9f1c920fdd01e2ba P a47e5c56f13843018ac584914a8d4707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:07.494303 7940 ts_tablet_manager.cc:1367] T a25f2855fb7d4eac9f1c920fdd01e2ba P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 20:46:07.494810 7943 tablet_bootstrap.cc:654] T a25f2855fb7d4eac9f1c920fdd01e2ba P 201d2b6485084e7dad2cbb722af94779: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:07.494786 7940 raft_consensus.cc:348] T a25f2855fb7d4eac9f1c920fdd01e2ba P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:07.494947 7940 raft_consensus.cc:374] T a25f2855fb7d4eac9f1c920fdd01e2ba P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:07.495024 7940 raft_consensus.cc:729] T a25f2855fb7d4eac9f1c920fdd01e2ba P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:46:07.495220 7940 consensus_queue.cc:260] T a25f2855fb7d4eac9f1c920fdd01e2ba P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:07.495582 7940 ts_tablet_manager.cc:1398] T a25f2855fb7d4eac9f1c920fdd01e2ba P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:07.495962 7939 ts_tablet_manager.cc:1398] T a25f2855fb7d4eac9f1c920fdd01e2ba P a47e5c56f13843018ac584914a8d4707: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 20:46:07.497161 7943 tablet_bootstrap.cc:492] T a25f2855fb7d4eac9f1c920fdd01e2ba P 201d2b6485084e7dad2cbb722af94779: No bootstrap required, opened a new log
I20240502 20:46:07.497253 7943 ts_tablet_manager.cc:1367] T a25f2855fb7d4eac9f1c920fdd01e2ba P 201d2b6485084e7dad2cbb722af94779: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:46:07.498064 7943 raft_consensus.cc:348] T a25f2855fb7d4eac9f1c920fdd01e2ba P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:07.498180 7943 raft_consensus.cc:374] T a25f2855fb7d4eac9f1c920fdd01e2ba P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:07.498222 7943 raft_consensus.cc:729] T a25f2855fb7d4eac9f1c920fdd01e2ba P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Initialized, Role: FOLLOWER
I20240502 20:46:07.498399 7943 consensus_queue.cc:260] T a25f2855fb7d4eac9f1c920fdd01e2ba P 201d2b6485084e7dad2cbb722af94779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:07.498682 7943 ts_tablet_manager.cc:1398] T a25f2855fb7d4eac9f1c920fdd01e2ba P 201d2b6485084e7dad2cbb722af94779: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:07.502687 7098 raft_consensus.cc:1203] T 8e7e145762bf458a8374bfc503025a17 P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:07.509687 7544 raft_consensus.cc:482] T a25f2855fb7d4eac9f1c920fdd01e2ba P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:07.509765 7544 raft_consensus.cc:504] T a25f2855fb7d4eac9f1c920fdd01e2ba P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:07.510732 7544 leader_election.cc:288] T a25f2855fb7d4eac9f1c920fdd01e2ba P 201d2b6485084e7dad2cbb722af94779 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
I20240502 20:46:07.511023 4377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a25f2855fb7d4eac9f1c920fdd01e2ba" candidate_uuid: "201d2b6485084e7dad2cbb722af94779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2" is_pre_election: true
I20240502 20:46:07.511152 4377 raft_consensus.cc:2454] T a25f2855fb7d4eac9f1c920fdd01e2ba P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 201d2b6485084e7dad2cbb722af94779 in term 0.
I20240502 20:46:07.511596 7035 leader_election.cc:302] T a25f2855fb7d4eac9f1c920fdd01e2ba P 201d2b6485084e7dad2cbb722af94779 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201d2b6485084e7dad2cbb722af94779, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:46:07.511837 7544 raft_consensus.cc:2781] T a25f2855fb7d4eac9f1c920fdd01e2ba P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:07.511904 7544 raft_consensus.cc:482] T a25f2855fb7d4eac9f1c920fdd01e2ba P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:07.511942 7544 raft_consensus.cc:3037] T a25f2855fb7d4eac9f1c920fdd01e2ba P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:07.512694 7544 raft_consensus.cc:504] T a25f2855fb7d4eac9f1c920fdd01e2ba P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:07.513011 7544 leader_election.cc:288] T a25f2855fb7d4eac9f1c920fdd01e2ba P 201d2b6485084e7dad2cbb722af94779 [CANDIDATE]: Term 1 election: Requested vote from peers c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
I20240502 20:46:07.513486 4377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a25f2855fb7d4eac9f1c920fdd01e2ba" candidate_uuid: "201d2b6485084e7dad2cbb722af94779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2"
I20240502 20:46:07.513597 4377 raft_consensus.cc:3037] T a25f2855fb7d4eac9f1c920fdd01e2ba P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:07.514557 4377 raft_consensus.cc:2454] T a25f2855fb7d4eac9f1c920fdd01e2ba P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 201d2b6485084e7dad2cbb722af94779 in term 1.
I20240502 20:46:07.514648 6957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a25f2855fb7d4eac9f1c920fdd01e2ba" candidate_uuid: "201d2b6485084e7dad2cbb722af94779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47e5c56f13843018ac584914a8d4707" is_pre_election: true
I20240502 20:46:07.514817 6957 raft_consensus.cc:2454] T a25f2855fb7d4eac9f1c920fdd01e2ba P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 201d2b6485084e7dad2cbb722af94779 in term 0.
I20240502 20:46:07.514921 7035 leader_election.cc:302] T a25f2855fb7d4eac9f1c920fdd01e2ba P 201d2b6485084e7dad2cbb722af94779 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201d2b6485084e7dad2cbb722af94779, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:46:07.515094 7544 raft_consensus.cc:2781] T a25f2855fb7d4eac9f1c920fdd01e2ba P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:07.515136 6957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a25f2855fb7d4eac9f1c920fdd01e2ba" candidate_uuid: "201d2b6485084e7dad2cbb722af94779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47e5c56f13843018ac584914a8d4707"
I20240502 20:46:07.515235 6957 raft_consensus.cc:3037] T a25f2855fb7d4eac9f1c920fdd01e2ba P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:07.515267 7544 raft_consensus.cc:686] T a25f2855fb7d4eac9f1c920fdd01e2ba P 201d2b6485084e7dad2cbb722af94779 [term 1 LEADER]: Becoming Leader. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Running, Role: LEADER
I20240502 20:46:07.515542 7544 consensus_queue.cc:237] T a25f2855fb7d4eac9f1c920fdd01e2ba P 201d2b6485084e7dad2cbb722af94779 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:07.515995 6957 raft_consensus.cc:2454] T a25f2855fb7d4eac9f1c920fdd01e2ba P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 201d2b6485084e7dad2cbb722af94779 in term 1.
I20240502 20:46:07.517410 4230 catalog_manager.cc:5484] T a25f2855fb7d4eac9f1c920fdd01e2ba P 201d2b6485084e7dad2cbb722af94779 reported cstate change: term changed from 0 to 1, leader changed from <none> to 201d2b6485084e7dad2cbb722af94779 (127.30.32.72). New cstate: current_term: 1 leader_uuid: "201d2b6485084e7dad2cbb722af94779" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:07.525833 7937 consensus_queue.cc:1035] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:07.529593 7432 consensus_queue.cc:1035] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:07.540808 7921 raft_consensus.cc:1050] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc: attempting to promote NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 to VOTER
I20240502 20:46:07.541326 7921 consensus_queue.cc:237] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } }
I20240502 20:46:07.542850 6831 raft_consensus.cc:1261] T d37730b109d04b6395f75c7552524877 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEARNER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:07.542995 4637 raft_consensus.cc:1261] T d37730b109d04b6395f75c7552524877 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:07.543269 4377 raft_consensus.cc:1261] T d37730b109d04b6395f75c7552524877 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:07.543278 6375 consensus_queue.cc:1035] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:07.543709 6375 consensus_queue.cc:1035] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:07.543880 7967 consensus_queue.cc:1035] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:07.545907 7967 raft_consensus.cc:2932] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:07.546823 4637 raft_consensus.cc:2932] T d37730b109d04b6395f75c7552524877 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:07.547976 4377 raft_consensus.cc:2932] T d37730b109d04b6395f75c7552524877 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:07.549454 6831 raft_consensus.cc:2932] T d37730b109d04b6395f75c7552524877 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:07.549986 6702 raft_consensus.cc:1203] T 00eb91e624554baf918e96b098019be8 P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:07.550562 4235 catalog_manager.cc:5484] T d37730b109d04b6395f75c7552524877 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index 4 to 5, a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:07.555410 4508 consensus_queue.cc:237] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } }
I20240502 20:46:07.556638 6831 raft_consensus.cc:1261] T d37730b109d04b6395f75c7552524877 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:07.557026 4637 raft_consensus.cc:1261] T d37730b109d04b6395f75c7552524877 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:07.557090 7921 consensus_queue.cc:1035] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:07.557425 6375 consensus_queue.cc:1035] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:07.559970 7967 raft_consensus.cc:2932] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:07.560475 6831 raft_consensus.cc:2932] T d37730b109d04b6395f75c7552524877 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:07.561525 4221 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d37730b109d04b6395f75c7552524877 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:07.562312 4637 raft_consensus.cc:2932] T d37730b109d04b6395f75c7552524877 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:07.562872 4228 catalog_manager.cc:5484] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: config changed from index 5 to 6, VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) evicted. New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:07.566164 4356 tablet_service.cc:1514] Processing DeleteTablet for tablet d37730b109d04b6395f75c7552524877 with delete_type TABLET_DATA_TOMBSTONED (TS c92e6e46782849479c8a0c6d82276fb2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38306
I20240502 20:46:07.566478 7297 tablet_replica.cc:330] T d37730b109d04b6395f75c7552524877 P c92e6e46782849479c8a0c6d82276fb2: stopping tablet replica
I20240502 20:46:07.566654 7297 raft_consensus.cc:2229] T d37730b109d04b6395f75c7552524877 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:07.566818 7297 raft_consensus.cc:2258] T d37730b109d04b6395f75c7552524877 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:07.567497 7297 ts_tablet_manager.cc:1875] T d37730b109d04b6395f75c7552524877 P c92e6e46782849479c8a0c6d82276fb2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:07.569615 7297 ts_tablet_manager.cc:1888] T d37730b109d04b6395f75c7552524877 P c92e6e46782849479c8a0c6d82276fb2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:46:07.569734 7297 log.cc:1196] T d37730b109d04b6395f75c7552524877 P c92e6e46782849479c8a0c6d82276fb2: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal/wals/d37730b109d04b6395f75c7552524877
I20240502 20:46:07.570240 4222 catalog_manager.cc:4830] TS c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827): tablet d37730b109d04b6395f75c7552524877 (table rebalance_test_table_7 [id=1e32a0fff569475cbd44b12c4fd5e8e6]) successfully deleted
I20240502 20:46:07.574527 6957 raft_consensus.cc:1203] T 8565ebbd21dd4f708a0254d1ed2d6670 P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:07.591477 6831 raft_consensus.cc:1203] T 02bddc1866e04858894e047ec100a6f0 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:07.610288 4229 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41812:
name: "rebalancer_extra_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:46:07.611349 6957 raft_consensus.cc:1203] T 428ffbac75db466191cb3bef63b4eb79 P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:07.617439 6683 tablet_service.cc:1467] Processing CreateTablet for tablet 3d938d01d5154590a9262acd54094ad2 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=1bde7089cdee45b68bdc73a007b74174]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:07.617549 6946 tablet_service.cc:1467] Processing CreateTablet for tablet 3d938d01d5154590a9262acd54094ad2 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=1bde7089cdee45b68bdc73a007b74174]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:07.617792 6683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d938d01d5154590a9262acd54094ad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:07.617866 6946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d938d01d5154590a9262acd54094ad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:07.619673 7939 tablet_bootstrap.cc:492] T 3d938d01d5154590a9262acd54094ad2 P a47e5c56f13843018ac584914a8d4707: Bootstrap starting.
I20240502 20:46:07.619855 6947 tablet_service.cc:1467] Processing CreateTablet for tablet 747e7331d7ad4dd4b132672254037e1b (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=1bde7089cdee45b68bdc73a007b74174]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:07.620046 6947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 747e7331d7ad4dd4b132672254037e1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:07.620528 7939 tablet_bootstrap.cc:654] T 3d938d01d5154590a9262acd54094ad2 P a47e5c56f13843018ac584914a8d4707: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:07.620644 6815 tablet_service.cc:1467] Processing CreateTablet for tablet 747e7331d7ad4dd4b132672254037e1b (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=1bde7089cdee45b68bdc73a007b74174]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:07.620644 6809 tablet_service.cc:1467] Processing CreateTablet for tablet 6ed088a162a14d37b5da21ebb468a03b (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=1bde7089cdee45b68bdc73a007b74174]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:07.621014 6815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 747e7331d7ad4dd4b132672254037e1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:07.621136 6809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ed088a162a14d37b5da21ebb468a03b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:07.622697 4356 tablet_service.cc:1467] Processing CreateTablet for tablet 6ed088a162a14d37b5da21ebb468a03b (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=1bde7089cdee45b68bdc73a007b74174]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:07.623011 4356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ed088a162a14d37b5da21ebb468a03b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:07.623135 7078 tablet_service.cc:1467] Processing CreateTablet for tablet 3d938d01d5154590a9262acd54094ad2 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=1bde7089cdee45b68bdc73a007b74174]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:07.623389 7078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d938d01d5154590a9262acd54094ad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:07.623397 7939 tablet_bootstrap.cc:492] T 3d938d01d5154590a9262acd54094ad2 P a47e5c56f13843018ac584914a8d4707: No bootstrap required, opened a new log
I20240502 20:46:07.623482 7939 ts_tablet_manager.cc:1367] T 3d938d01d5154590a9262acd54094ad2 P a47e5c56f13843018ac584914a8d4707: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240502 20:46:07.623991 7939 raft_consensus.cc:348] T 3d938d01d5154590a9262acd54094ad2 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:07.624096 7939 raft_consensus.cc:374] T 3d938d01d5154590a9262acd54094ad2 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:07.624143 7939 raft_consensus.cc:729] T 3d938d01d5154590a9262acd54094ad2 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Initialized, Role: FOLLOWER
I20240502 20:46:07.624277 7939 consensus_queue.cc:260] T 3d938d01d5154590a9262acd54094ad2 P a47e5c56f13843018ac584914a8d4707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:07.624555 7939 ts_tablet_manager.cc:1398] T 3d938d01d5154590a9262acd54094ad2 P a47e5c56f13843018ac584914a8d4707: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:07.624715 7939 tablet_bootstrap.cc:492] T 747e7331d7ad4dd4b132672254037e1b P a47e5c56f13843018ac584914a8d4707: Bootstrap starting.
I20240502 20:46:07.624840 4229 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 20:46:07.625617 7939 tablet_bootstrap.cc:654] T 747e7331d7ad4dd4b132672254037e1b P a47e5c56f13843018ac584914a8d4707: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:07.625686 7079 tablet_service.cc:1467] Processing CreateTablet for tablet 6ed088a162a14d37b5da21ebb468a03b (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=1bde7089cdee45b68bdc73a007b74174]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:07.625767 7940 tablet_bootstrap.cc:492] T 6ed088a162a14d37b5da21ebb468a03b P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:46:07.625907 7079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ed088a162a14d37b5da21ebb468a03b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:07.626677 7940 tablet_bootstrap.cc:654] T 6ed088a162a14d37b5da21ebb468a03b P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:07.626888 6683 tablet_service.cc:1467] Processing CreateTablet for tablet 747e7331d7ad4dd4b132672254037e1b (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=1bde7089cdee45b68bdc73a007b74174]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:07.627154 6683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 747e7331d7ad4dd4b132672254037e1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:07.627879 7943 tablet_bootstrap.cc:492] T 3d938d01d5154590a9262acd54094ad2 P 201d2b6485084e7dad2cbb722af94779: Bootstrap starting.
I20240502 20:46:07.628005 7940 tablet_bootstrap.cc:492] T 6ed088a162a14d37b5da21ebb468a03b P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:46:07.628073 7940 ts_tablet_manager.cc:1367] T 6ed088a162a14d37b5da21ebb468a03b P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:07.628533 7940 raft_consensus.cc:348] T 6ed088a162a14d37b5da21ebb468a03b P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:07.628643 7940 raft_consensus.cc:374] T 6ed088a162a14d37b5da21ebb468a03b P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:07.628681 7940 raft_consensus.cc:729] T 6ed088a162a14d37b5da21ebb468a03b P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:46:07.628791 7943 tablet_bootstrap.cc:654] T 3d938d01d5154590a9262acd54094ad2 P 201d2b6485084e7dad2cbb722af94779: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:07.628808 7940 consensus_queue.cc:260] T 6ed088a162a14d37b5da21ebb468a03b P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:07.629179 7940 ts_tablet_manager.cc:1398] T 6ed088a162a14d37b5da21ebb468a03b P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:07.631543 7943 tablet_bootstrap.cc:492] T 3d938d01d5154590a9262acd54094ad2 P 201d2b6485084e7dad2cbb722af94779: No bootstrap required, opened a new log
I20240502 20:46:07.631639 7943 ts_tablet_manager.cc:1367] T 3d938d01d5154590a9262acd54094ad2 P 201d2b6485084e7dad2cbb722af94779: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 20:46:07.632090 7943 raft_consensus.cc:348] T 3d938d01d5154590a9262acd54094ad2 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:07.632181 7943 raft_consensus.cc:374] T 3d938d01d5154590a9262acd54094ad2 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:07.632215 7943 raft_consensus.cc:729] T 3d938d01d5154590a9262acd54094ad2 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Initialized, Role: FOLLOWER
I20240502 20:46:07.632345 7943 consensus_queue.cc:260] T 3d938d01d5154590a9262acd54094ad2 P 201d2b6485084e7dad2cbb722af94779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:07.632652 7943 ts_tablet_manager.cc:1398] T 3d938d01d5154590a9262acd54094ad2 P 201d2b6485084e7dad2cbb722af94779: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:07.632810 7943 tablet_bootstrap.cc:492] T 6ed088a162a14d37b5da21ebb468a03b P 201d2b6485084e7dad2cbb722af94779: Bootstrap starting.
I20240502 20:46:07.633533 8014 tablet_bootstrap.cc:492] T 747e7331d7ad4dd4b132672254037e1b P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap starting.
I20240502 20:46:07.634351 7943 tablet_bootstrap.cc:654] T 6ed088a162a14d37b5da21ebb468a03b P 201d2b6485084e7dad2cbb722af94779: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:07.635076 8014 tablet_bootstrap.cc:654] T 747e7331d7ad4dd4b132672254037e1b P a4bf1df7fa6b4948be502c182bfbb833: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:07.638396 8014 tablet_bootstrap.cc:492] T 747e7331d7ad4dd4b132672254037e1b P a4bf1df7fa6b4948be502c182bfbb833: No bootstrap required, opened a new log
I20240502 20:46:07.638723 4377 raft_consensus.cc:1261] T a25f2855fb7d4eac9f1c920fdd01e2ba P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer 201d2b6485084e7dad2cbb722af94779: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:07.639011 8014 ts_tablet_manager.cc:1367] T 747e7331d7ad4dd4b132672254037e1b P a4bf1df7fa6b4948be502c182bfbb833: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 20:46:07.639081 6957 raft_consensus.cc:1261] T a25f2855fb7d4eac9f1c920fdd01e2ba P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Refusing update from remote peer 201d2b6485084e7dad2cbb722af94779: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:07.639317 7544 consensus_queue.cc:1035] T a25f2855fb7d4eac9f1c920fdd01e2ba P 201d2b6485084e7dad2cbb722af94779 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:07.639680 7544 consensus_queue.cc:1035] T a25f2855fb7d4eac9f1c920fdd01e2ba P 201d2b6485084e7dad2cbb722af94779 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:07.639513 8014 raft_consensus.cc:348] T 747e7331d7ad4dd4b132672254037e1b P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:07.640795 8014 raft_consensus.cc:374] T 747e7331d7ad4dd4b132672254037e1b P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:07.640842 8014 raft_consensus.cc:729] T 747e7331d7ad4dd4b132672254037e1b P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Initialized, Role: FOLLOWER
I20240502 20:46:07.641017 8014 consensus_queue.cc:260] T 747e7331d7ad4dd4b132672254037e1b P a4bf1df7fa6b4948be502c182bfbb833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:07.642441 4228 catalog_manager.cc:5576] TS a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841) does not have the latest schema for tablet a25f2855fb7d4eac9f1c920fdd01e2ba (table rebalancer_dynamic_table [id=0d23318bd0f5482face4d225e00e7757]). Expected version 1 got 0
I20240502 20:46:07.643014 4233 catalog_manager.cc:5576] TS c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827) does not have the latest schema for tablet a25f2855fb7d4eac9f1c920fdd01e2ba (table rebalancer_dynamic_table [id=0d23318bd0f5482face4d225e00e7757]). Expected version 1 got 0
I20240502 20:46:07.643070 8014 ts_tablet_manager.cc:1398] T 747e7331d7ad4dd4b132672254037e1b P a4bf1df7fa6b4948be502c182bfbb833: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 20:46:07.643267 8014 tablet_bootstrap.cc:492] T 6ed088a162a14d37b5da21ebb468a03b P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap starting.
I20240502 20:46:07.644749 7881 mvcc.cc:205] Tried to move back new op lower bound from 7023340616242032640 to 7023340615745007616. Current Snapshot: MvccSnapshot[applied={T|T < 7023340616242032640}]
I20240502 20:46:07.646311 8012 tablet_bootstrap.cc:492] T 3d938d01d5154590a9262acd54094ad2 P 7845c397230d49358ba2fad9db53f97c: Bootstrap starting.
I20240502 20:46:07.646612 8027 tablet.cc:1722] T a25f2855fb7d4eac9f1c920fdd01e2ba P 201d2b6485084e7dad2cbb722af94779: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 20:46:07.647243 8014 tablet_bootstrap.cc:654] T 6ed088a162a14d37b5da21ebb468a03b P a4bf1df7fa6b4948be502c182bfbb833: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:07.647611 4233 catalog_manager.cc:5576] TS 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643) does not have the latest schema for tablet a25f2855fb7d4eac9f1c920fdd01e2ba (table rebalancer_dynamic_table [id=0d23318bd0f5482face4d225e00e7757]). Expected version 1 got 0
I20240502 20:46:07.648254 8029 tablet.cc:1722] T a25f2855fb7d4eac9f1c920fdd01e2ba P c92e6e46782849479c8a0c6d82276fb2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 20:46:07.648767 4228 catalog_manager.cc:5576] TS c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827) does not have the latest schema for tablet a25f2855fb7d4eac9f1c920fdd01e2ba (table rebalancer_dynamic_table [id=0d23318bd0f5482face4d225e00e7757]). Expected version 1 got 0
I20240502 20:46:07.649278 4233 catalog_manager.cc:5576] TS a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841) does not have the latest schema for tablet a25f2855fb7d4eac9f1c920fdd01e2ba (table rebalancer_dynamic_table [id=0d23318bd0f5482face4d225e00e7757]). Expected version 1 got 0
I20240502 20:46:07.649724 8029 tablet.cc:1620] T a25f2855fb7d4eac9f1c920fdd01e2ba P c92e6e46782849479c8a0c6d82276fb2: MemRowSet was empty: no flush needed.
I20240502 20:46:07.650300 7939 tablet_bootstrap.cc:492] T 747e7331d7ad4dd4b132672254037e1b P a47e5c56f13843018ac584914a8d4707: No bootstrap required, opened a new log
I20240502 20:46:07.650391 7939 ts_tablet_manager.cc:1367] T 747e7331d7ad4dd4b132672254037e1b P a47e5c56f13843018ac584914a8d4707: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.000s
I20240502 20:46:07.650854 7939 raft_consensus.cc:348] T 747e7331d7ad4dd4b132672254037e1b P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:07.650975 7939 raft_consensus.cc:374] T 747e7331d7ad4dd4b132672254037e1b P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:07.651008 7939 raft_consensus.cc:729] T 747e7331d7ad4dd4b132672254037e1b P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Initialized, Role: FOLLOWER
I20240502 20:46:07.651126 7939 consensus_queue.cc:260] T 747e7331d7ad4dd4b132672254037e1b P a47e5c56f13843018ac584914a8d4707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:07.651438 7939 ts_tablet_manager.cc:1398] T 747e7331d7ad4dd4b132672254037e1b P a47e5c56f13843018ac584914a8d4707: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:07.651789 6957 raft_consensus.cc:1203] T f09dcfea3eb347e687f3c72f4ccc6929 P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:07.652936 8014 tablet_bootstrap.cc:492] T 6ed088a162a14d37b5da21ebb468a03b P a4bf1df7fa6b4948be502c182bfbb833: No bootstrap required, opened a new log
I20240502 20:46:07.653074 8014 ts_tablet_manager.cc:1367] T 6ed088a162a14d37b5da21ebb468a03b P a4bf1df7fa6b4948be502c182bfbb833: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240502 20:46:07.653754 8014 raft_consensus.cc:348] T 6ed088a162a14d37b5da21ebb468a03b P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:07.653882 8014 raft_consensus.cc:374] T 6ed088a162a14d37b5da21ebb468a03b P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:07.653934 8014 raft_consensus.cc:729] T 6ed088a162a14d37b5da21ebb468a03b P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Initialized, Role: FOLLOWER
I20240502 20:46:07.654081 8014 consensus_queue.cc:260] T 6ed088a162a14d37b5da21ebb468a03b P a4bf1df7fa6b4948be502c182bfbb833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:07.654490 8014 ts_tablet_manager.cc:1398] T 6ed088a162a14d37b5da21ebb468a03b P a4bf1df7fa6b4948be502c182bfbb833: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:07.655447 8012 tablet_bootstrap.cc:654] T 3d938d01d5154590a9262acd54094ad2 P 7845c397230d49358ba2fad9db53f97c: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:07.649169 7943 tablet_bootstrap.cc:492] T 6ed088a162a14d37b5da21ebb468a03b P 201d2b6485084e7dad2cbb722af94779: No bootstrap required, opened a new log
I20240502 20:46:07.656162 7943 ts_tablet_manager.cc:1367] T 6ed088a162a14d37b5da21ebb468a03b P 201d2b6485084e7dad2cbb722af94779: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.000s
I20240502 20:46:07.656607 7943 raft_consensus.cc:348] T 6ed088a162a14d37b5da21ebb468a03b P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:07.656749 7943 raft_consensus.cc:374] T 6ed088a162a14d37b5da21ebb468a03b P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:07.656785 7943 raft_consensus.cc:729] T 6ed088a162a14d37b5da21ebb468a03b P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Initialized, Role: FOLLOWER
I20240502 20:46:07.656905 7943 consensus_queue.cc:260] T 6ed088a162a14d37b5da21ebb468a03b P 201d2b6485084e7dad2cbb722af94779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:07.657230 7943 ts_tablet_manager.cc:1398] T 6ed088a162a14d37b5da21ebb468a03b P 201d2b6485084e7dad2cbb722af94779: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:07.656610 8027 tablet.cc:1620] T a25f2855fb7d4eac9f1c920fdd01e2ba P 201d2b6485084e7dad2cbb722af94779: MemRowSet was empty: no flush needed.
I20240502 20:46:07.658152 4233 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: rebalancer_dynamic_table [id=0d23318bd0f5482face4d225e00e7757] alter complete (version 1)
I20240502 20:46:07.658146 6702 raft_consensus.cc:1203] T d37fab152a8a44868d3160ec3057bb98 P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:07.658931 8012 tablet_bootstrap.cc:492] T 3d938d01d5154590a9262acd54094ad2 P 7845c397230d49358ba2fad9db53f97c: No bootstrap required, opened a new log
I20240502 20:46:07.659039 8012 ts_tablet_manager.cc:1367] T 3d938d01d5154590a9262acd54094ad2 P 7845c397230d49358ba2fad9db53f97c: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240502 20:46:07.659502 8012 raft_consensus.cc:348] T 3d938d01d5154590a9262acd54094ad2 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:07.659673 8012 raft_consensus.cc:374] T 3d938d01d5154590a9262acd54094ad2 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:07.659745 8012 raft_consensus.cc:729] T 3d938d01d5154590a9262acd54094ad2 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Initialized, Role: FOLLOWER
I20240502 20:46:07.659916 8012 consensus_queue.cc:260] T 3d938d01d5154590a9262acd54094ad2 P 7845c397230d49358ba2fad9db53f97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:07.660352 8012 ts_tablet_manager.cc:1398] T 3d938d01d5154590a9262acd54094ad2 P 7845c397230d49358ba2fad9db53f97c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:07.660523 8012 tablet_bootstrap.cc:492] T 747e7331d7ad4dd4b132672254037e1b P 7845c397230d49358ba2fad9db53f97c: Bootstrap starting.
I20240502 20:46:07.661093 8035 tablet.cc:1722] T a25f2855fb7d4eac9f1c920fdd01e2ba P a47e5c56f13843018ac584914a8d4707: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 20:46:07.661455 8012 tablet_bootstrap.cc:654] T 747e7331d7ad4dd4b132672254037e1b P 7845c397230d49358ba2fad9db53f97c: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:07.661489 8035 tablet.cc:1620] T a25f2855fb7d4eac9f1c920fdd01e2ba P a47e5c56f13843018ac584914a8d4707: MemRowSet was empty: no flush needed.
I20240502 20:46:07.666726 8012 tablet_bootstrap.cc:492] T 747e7331d7ad4dd4b132672254037e1b P 7845c397230d49358ba2fad9db53f97c: No bootstrap required, opened a new log
I20240502 20:46:07.666873 8012 ts_tablet_manager.cc:1367] T 747e7331d7ad4dd4b132672254037e1b P 7845c397230d49358ba2fad9db53f97c: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 20:46:07.667768 8012 raft_consensus.cc:348] T 747e7331d7ad4dd4b132672254037e1b P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:07.667886 8012 raft_consensus.cc:374] T 747e7331d7ad4dd4b132672254037e1b P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:07.667945 8012 raft_consensus.cc:729] T 747e7331d7ad4dd4b132672254037e1b P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Initialized, Role: FOLLOWER
I20240502 20:46:07.668087 8012 consensus_queue.cc:260] T 747e7331d7ad4dd4b132672254037e1b P 7845c397230d49358ba2fad9db53f97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:07.668408 8012 ts_tablet_manager.cc:1398] T 747e7331d7ad4dd4b132672254037e1b P 7845c397230d49358ba2fad9db53f97c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:07.685727 6831 raft_consensus.cc:1203] T 61f8c8d2925d46bdad8e903b48c3046e P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:07.687826 8026 raft_consensus.cc:482] T 6ed088a162a14d37b5da21ebb468a03b P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:07.687958 8026 raft_consensus.cc:504] T 6ed088a162a14d37b5da21ebb468a03b P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:07.688482 8026 leader_election.cc:288] T 6ed088a162a14d37b5da21ebb468a03b P 201d2b6485084e7dad2cbb722af94779 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827), a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867)
I20240502 20:46:07.689373 4377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ed088a162a14d37b5da21ebb468a03b" candidate_uuid: "201d2b6485084e7dad2cbb722af94779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2" is_pre_election: true
I20240502 20:46:07.689510 4377 raft_consensus.cc:2454] T 6ed088a162a14d37b5da21ebb468a03b P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 201d2b6485084e7dad2cbb722af94779 in term 0.
I20240502 20:46:07.689795 7035 leader_election.cc:302] T 6ed088a162a14d37b5da21ebb468a03b P 201d2b6485084e7dad2cbb722af94779 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201d2b6485084e7dad2cbb722af94779, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:46:07.690063 8026 raft_consensus.cc:2781] T 6ed088a162a14d37b5da21ebb468a03b P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:07.690130 8026 raft_consensus.cc:482] T 6ed088a162a14d37b5da21ebb468a03b P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:07.690164 8026 raft_consensus.cc:3037] T 6ed088a162a14d37b5da21ebb468a03b P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:07.691205 8026 raft_consensus.cc:504] T 6ed088a162a14d37b5da21ebb468a03b P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:07.691540 8026 leader_election.cc:288] T 6ed088a162a14d37b5da21ebb468a03b P 201d2b6485084e7dad2cbb722af94779 [CANDIDATE]: Term 1 election: Requested vote from peers c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827), a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867)
I20240502 20:46:07.692445 4374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ed088a162a14d37b5da21ebb468a03b" candidate_uuid: "201d2b6485084e7dad2cbb722af94779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2"
I20240502 20:46:07.692585 4374 raft_consensus.cc:3037] T 6ed088a162a14d37b5da21ebb468a03b P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:07.693504 4374 raft_consensus.cc:2454] T 6ed088a162a14d37b5da21ebb468a03b P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 201d2b6485084e7dad2cbb722af94779 in term 1.
I20240502 20:46:07.693812 7035 leader_election.cc:302] T 6ed088a162a14d37b5da21ebb468a03b P 201d2b6485084e7dad2cbb722af94779 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201d2b6485084e7dad2cbb722af94779, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:46:07.694085 8026 raft_consensus.cc:2781] T 6ed088a162a14d37b5da21ebb468a03b P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:07.694180 8026 raft_consensus.cc:686] T 6ed088a162a14d37b5da21ebb468a03b P 201d2b6485084e7dad2cbb722af94779 [term 1 LEADER]: Becoming Leader. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Running, Role: LEADER
I20240502 20:46:07.694319 8026 consensus_queue.cc:237] T 6ed088a162a14d37b5da21ebb468a03b P 201d2b6485084e7dad2cbb722af94779 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:07.696020 4228 catalog_manager.cc:5484] T 6ed088a162a14d37b5da21ebb468a03b P 201d2b6485084e7dad2cbb722af94779 reported cstate change: term changed from 0 to 1, leader changed from <none> to 201d2b6485084e7dad2cbb722af94779 (127.30.32.72). New cstate: current_term: 1 leader_uuid: "201d2b6485084e7dad2cbb722af94779" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } health_report { overall_health: HEALTHY } } }
I20240502 20:46:07.700047 6831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ed088a162a14d37b5da21ebb468a03b" candidate_uuid: "201d2b6485084e7dad2cbb722af94779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833" is_pre_election: true
I20240502 20:46:07.700228 6831 raft_consensus.cc:2454] T 6ed088a162a14d37b5da21ebb468a03b P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 201d2b6485084e7dad2cbb722af94779 in term 0.
I20240502 20:46:07.700456 6835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ed088a162a14d37b5da21ebb468a03b" candidate_uuid: "201d2b6485084e7dad2cbb722af94779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833"
I20240502 20:46:07.700564 6835 raft_consensus.cc:3037] T 6ed088a162a14d37b5da21ebb468a03b P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:07.701694 6835 raft_consensus.cc:2454] T 6ed088a162a14d37b5da21ebb468a03b P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 201d2b6485084e7dad2cbb722af94779 in term 1.
I20240502 20:46:07.737509 7404 raft_consensus.cc:1050] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc: attempting to promote NON_VOTER 201d2b6485084e7dad2cbb722af94779 to VOTER
I20240502 20:46:07.737998 7404 consensus_queue.cc:237] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } }
I20240502 20:46:07.739836 4637 raft_consensus.cc:1261] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:07.740223 7098 raft_consensus.cc:1261] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:07.740244 4374 raft_consensus.cc:1261] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:07.740609 7967 consensus_queue.cc:1035] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:07.740777 7921 consensus_queue.cc:1035] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:07.741412 7318 raft_consensus.cc:1050] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd: attempting to promote NON_VOTER 7845c397230d49358ba2fad9db53f97c to VOTER
I20240502 20:46:07.741539 7967 consensus_queue.cc:1035] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:07.741915 7318 consensus_queue.cc:237] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } }
I20240502 20:46:07.743193 6702 raft_consensus.cc:1261] T a2e81d2c63cd4e4cac5086506b5c4a12 P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:07.743357 4638 raft_consensus.cc:1261] T a2e81d2c63cd4e4cac5086506b5c4a12 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:07.743813 7996 consensus_queue.cc:1035] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:07.744242 4508 raft_consensus.cc:1261] T a2e81d2c63cd4e4cac5086506b5c4a12 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:07.744455 6241 consensus_queue.cc:1035] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:07.744626 7318 consensus_queue.cc:1035] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:07.749050 7318 raft_consensus.cc:2932] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:07.749846 7967 raft_consensus.cc:2932] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:07.751755 4508 raft_consensus.cc:2932] T a2e81d2c63cd4e4cac5086506b5c4a12 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:07.752521 4638 raft_consensus.cc:2932] T a2e81d2c63cd4e4cac5086506b5c4a12 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:07.752935 4374 raft_consensus.cc:2932] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:07.753240 4229 catalog_manager.cc:5484] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd reported cstate change: config changed from index 4 to 5, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:07.753485 6702 raft_consensus.cc:2932] T a2e81d2c63cd4e4cac5086506b5c4a12 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:07.754923 7165 raft_consensus.cc:482] T 3d938d01d5154590a9262acd54094ad2 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:07.755026 7165 raft_consensus.cc:504] T 3d938d01d5154590a9262acd54094ad2 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:07.756078 7165 leader_election.cc:288] T 3d938d01d5154590a9262acd54094ad2 P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
I20240502 20:46:07.757289 4233 catalog_manager.cc:5484] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: config changed from index 4 to 5, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:07.758144 7098 raft_consensus.cc:2932] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:07.761766 4230 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 20:46:07.762713 6957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d938d01d5154590a9262acd54094ad2" candidate_uuid: "7845c397230d49358ba2fad9db53f97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47e5c56f13843018ac584914a8d4707" is_pre_election: true
I20240502 20:46:07.762861 6957 raft_consensus.cc:2454] T 3d938d01d5154590a9262acd54094ad2 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7845c397230d49358ba2fad9db53f97c in term 0.
I20240502 20:46:07.763445 6639 leader_election.cc:302] T 3d938d01d5154590a9262acd54094ad2 P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7845c397230d49358ba2fad9db53f97c, a47e5c56f13843018ac584914a8d4707; no voters:
I20240502 20:46:07.763610 4768 consensus_queue.cc:237] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } }
I20240502 20:46:07.764811 7098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d938d01d5154590a9262acd54094ad2" candidate_uuid: "7845c397230d49358ba2fad9db53f97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201d2b6485084e7dad2cbb722af94779" is_pre_election: true
I20240502 20:46:07.764952 7098 raft_consensus.cc:2454] T 3d938d01d5154590a9262acd54094ad2 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7845c397230d49358ba2fad9db53f97c in term 0.
I20240502 20:46:07.765935 7165 raft_consensus.cc:2781] T 3d938d01d5154590a9262acd54094ad2 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:07.765981 6702 raft_consensus.cc:1261] T a2e81d2c63cd4e4cac5086506b5c4a12 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:07.766302 4508 raft_consensus.cc:1261] T a2e81d2c63cd4e4cac5086506b5c4a12 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:07.766000 7165 raft_consensus.cc:482] T 3d938d01d5154590a9262acd54094ad2 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:07.766739 7165 raft_consensus.cc:3037] T 3d938d01d5154590a9262acd54094ad2 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:07.767040 7318 consensus_queue.cc:1035] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:07.767354 7318 consensus_queue.cc:1035] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:07.767792 7165 raft_consensus.cc:504] T 3d938d01d5154590a9262acd54094ad2 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:07.768170 7165 leader_election.cc:288] T 3d938d01d5154590a9262acd54094ad2 P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 election: Requested vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
I20240502 20:46:07.769821 4637 raft_consensus.cc:2932] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:07.770175 7318 raft_consensus.cc:2932] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:07.772017 4222 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a2e81d2c63cd4e4cac5086506b5c4a12 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:07.772281 8027 tablet.cc:1722] T a25f2855fb7d4eac9f1c920fdd01e2ba P 201d2b6485084e7dad2cbb722af94779: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 20:46:07.772661 8027 tablet.cc:1620] T a25f2855fb7d4eac9f1c920fdd01e2ba P 201d2b6485084e7dad2cbb722af94779: MemRowSet was empty: no flush needed.
I20240502 20:46:07.773052 4508 raft_consensus.cc:2932] T a2e81d2c63cd4e4cac5086506b5c4a12 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:07.774091 8035 tablet.cc:1722] T a25f2855fb7d4eac9f1c920fdd01e2ba P a47e5c56f13843018ac584914a8d4707: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 20:46:07.774366 8035 tablet.cc:1620] T a25f2855fb7d4eac9f1c920fdd01e2ba P a47e5c56f13843018ac584914a8d4707: MemRowSet was empty: no flush needed.
I20240502 20:46:07.776249 7098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d938d01d5154590a9262acd54094ad2" candidate_uuid: "7845c397230d49358ba2fad9db53f97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201d2b6485084e7dad2cbb722af94779"
I20240502 20:46:07.776253 8029 tablet.cc:1722] T a25f2855fb7d4eac9f1c920fdd01e2ba P c92e6e46782849479c8a0c6d82276fb2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 20:46:07.776363 7098 raft_consensus.cc:3037] T 3d938d01d5154590a9262acd54094ad2 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:07.776571 8029 tablet.cc:1620] T a25f2855fb7d4eac9f1c920fdd01e2ba P c92e6e46782849479c8a0c6d82276fb2: MemRowSet was empty: no flush needed.
I20240502 20:46:07.777256 7098 raft_consensus.cc:2454] T 3d938d01d5154590a9262acd54094ad2 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7845c397230d49358ba2fad9db53f97c in term 1.
I20240502 20:46:07.777607 6636 leader_election.cc:302] T 3d938d01d5154590a9262acd54094ad2 P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201d2b6485084e7dad2cbb722af94779, 7845c397230d49358ba2fad9db53f97c; no voters:
I20240502 20:46:07.779217 4228 catalog_manager.cc:5484] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd reported cstate change: config changed from index 5 to 6, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:07.779781 6957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d938d01d5154590a9262acd54094ad2" candidate_uuid: "7845c397230d49358ba2fad9db53f97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47e5c56f13843018ac584914a8d4707"
I20240502 20:46:07.779918 6957 raft_consensus.cc:3037] T 3d938d01d5154590a9262acd54094ad2 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:07.781260 6702 raft_consensus.cc:2932] T a2e81d2c63cd4e4cac5086506b5c4a12 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:07.783046 8116 raft_consensus.cc:2781] T 3d938d01d5154590a9262acd54094ad2 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:07.783254 8116 raft_consensus.cc:686] T 3d938d01d5154590a9262acd54094ad2 P 7845c397230d49358ba2fad9db53f97c [term 1 LEADER]: Becoming Leader. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Running, Role: LEADER
I20240502 20:46:07.783512 8116 consensus_queue.cc:237] T 3d938d01d5154590a9262acd54094ad2 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:07.786458 4232 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: rebalancer_dynamic_table [id=0d23318bd0f5482face4d225e00e7757] alter complete (version 2)
I20240502 20:46:07.786643 4618 tablet_service.cc:1514] Processing DeleteTablet for tablet a2e81d2c63cd4e4cac5086506b5c4a12 with delete_type TABLET_DATA_TOMBSTONED (TS 923cfab926e7443f8ffd7b21ad0a9d91 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57068
I20240502 20:46:07.786926 7913 tablet_replica.cc:330] T a2e81d2c63cd4e4cac5086506b5c4a12 P 923cfab926e7443f8ffd7b21ad0a9d91: stopping tablet replica
I20240502 20:46:07.787106 7913 raft_consensus.cc:2229] T a2e81d2c63cd4e4cac5086506b5c4a12 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:07.787303 7913 raft_consensus.cc:2258] T a2e81d2c63cd4e4cac5086506b5c4a12 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:07.788064 7913 ts_tablet_manager.cc:1875] T a2e81d2c63cd4e4cac5086506b5c4a12 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:07.790019 7913 ts_tablet_manager.cc:1888] T a2e81d2c63cd4e4cac5086506b5c4a12 P 923cfab926e7443f8ffd7b21ad0a9d91: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:46:07.790095 7913 log.cc:1196] T a2e81d2c63cd4e4cac5086506b5c4a12 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal/wals/a2e81d2c63cd4e4cac5086506b5c4a12
I20240502 20:46:07.790616 4219 catalog_manager.cc:4830] TS 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141): tablet a2e81d2c63cd4e4cac5086506b5c4a12 (table rebalance_test_table_1 [id=3a26d60c6de746e891d970c2ce9c55cc]) successfully deleted
I20240502 20:46:07.790798 4228 catalog_manager.cc:5484] T 3d938d01d5154590a9262acd54094ad2 P 7845c397230d49358ba2fad9db53f97c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7845c397230d49358ba2fad9db53f97c (127.30.32.69). New cstate: current_term: 1 leader_uuid: "7845c397230d49358ba2fad9db53f97c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:07.792419 7921 raft_consensus.cc:1050] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc: attempting to promote NON_VOTER 7845c397230d49358ba2fad9db53f97c to VOTER
I20240502 20:46:07.792889 7921 consensus_queue.cc:237] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } }
I20240502 20:46:07.794381 4768 raft_consensus.cc:1261] T d6fb394fdcff436886fe0634d79c024c P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:07.794431 6702 raft_consensus.cc:1261] T d6fb394fdcff436886fe0634d79c024c P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:07.794684 4637 raft_consensus.cc:1261] T d6fb394fdcff436886fe0634d79c024c P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:07.794890 6375 consensus_queue.cc:1035] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:07.795423 6375 consensus_queue.cc:1035] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:07.795964 6375 consensus_queue.cc:1035] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:07.796512 4231 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240502 20:46:07.796625 4231 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240502 20:46:07.800834 4231 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: Sending DeleteTablet for 4 replicas of tablet 28029d4e66d1453cbde9811ffc2a3283
I20240502 20:46:07.801271 4231 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: Sending DeleteTablet for 4 replicas of tablet 47c5087093214fbeae35590b4b3c3d42
I20240502 20:46:07.801544 4231 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: Sending DeleteTablet for 4 replicas of tablet 02bddc1866e04858894e047ec100a6f0
I20240502 20:46:07.801553 6375 raft_consensus.cc:2932] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:07.801891 4356 tablet_service.cc:1514] Processing DeleteTablet for tablet 28029d4e66d1453cbde9811ffc2a3283 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:07 UTC) from {username='slave'} at 127.0.0.1:38306
I20240502 20:46:07.802176 7079 tablet_service.cc:1514] Processing DeleteTablet for tablet 28029d4e66d1453cbde9811ffc2a3283 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:07 UTC) from {username='slave'} at 127.0.0.1:59162
I20240502 20:46:07.802402 4768 raft_consensus.cc:2932] T d6fb394fdcff436886fe0634d79c024c P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:07.802596 7297 tablet_replica.cc:330] T 28029d4e66d1453cbde9811ffc2a3283 P c92e6e46782849479c8a0c6d82276fb2: stopping tablet replica
I20240502 20:46:07.802764 7297 raft_consensus.cc:2229] T 28029d4e66d1453cbde9811ffc2a3283 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:07.802937 7297 raft_consensus.cc:2258] T 28029d4e66d1453cbde9811ffc2a3283 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:07.803493 7297 ts_tablet_manager.cc:1875] T 28029d4e66d1453cbde9811ffc2a3283 P c92e6e46782849479c8a0c6d82276fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:07.804186 6702 raft_consensus.cc:2932] T d6fb394fdcff436886fe0634d79c024c P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:07.804399 6957 raft_consensus.cc:2454] T 3d938d01d5154590a9262acd54094ad2 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7845c397230d49358ba2fad9db53f97c in term 1.
I20240502 20:46:07.805891 4637 raft_consensus.cc:2932] T d6fb394fdcff436886fe0634d79c024c P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:07.806552 6947 tablet_service.cc:1514] Processing DeleteTablet for tablet 47c5087093214fbeae35590b4b3c3d42 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:07 UTC) from {username='slave'} at 127.0.0.1:45156
I20240502 20:46:07.807318 4618 tablet_service.cc:1514] Processing DeleteTablet for tablet 28029d4e66d1453cbde9811ffc2a3283 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:07 UTC) from {username='slave'} at 127.0.0.1:57068
I20240502 20:46:07.807341 4228 catalog_manager.cc:5484] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: config changed from index 4 to 5, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:07.807507 6809 tablet_service.cc:1514] Processing DeleteTablet for tablet 02bddc1866e04858894e047ec100a6f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:07 UTC) from {username='slave'} at 127.0.0.1:35290
I20240502 20:46:07.807734 7908 tablet_replica.cc:330] T 02bddc1866e04858894e047ec100a6f0 P a4bf1df7fa6b4948be502c182bfbb833: stopping tablet replica
I20240502 20:46:07.807879 7908 raft_consensus.cc:2229] T 02bddc1866e04858894e047ec100a6f0 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEARNER]: Raft consensus shutting down.
I20240502 20:46:07.807951 4749 tablet_service.cc:1514] Processing DeleteTablet for tablet 47c5087093214fbeae35590b4b3c3d42 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:07 UTC) from {username='slave'} at 127.0.0.1:38526
I20240502 20:46:07.807991 7908 raft_consensus.cc:2258] T 02bddc1866e04858894e047ec100a6f0 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEARNER]: Raft consensus is shut down!
I20240502 20:46:07.808344 4750 tablet_service.cc:1514] Processing DeleteTablet for tablet 02bddc1866e04858894e047ec100a6f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:07 UTC) from {username='slave'} at 127.0.0.1:38526
I20240502 20:46:07.808490 7908 ts_tablet_manager.cc:1875] T 02bddc1866e04858894e047ec100a6f0 P a4bf1df7fa6b4948be502c182bfbb833: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:07.808514 7913 tablet_replica.cc:330] T 28029d4e66d1453cbde9811ffc2a3283 P 923cfab926e7443f8ffd7b21ad0a9d91: stopping tablet replica
I20240502 20:46:07.808676 7913 raft_consensus.cc:2229] T 28029d4e66d1453cbde9811ffc2a3283 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:07.808822 7913 raft_consensus.cc:2258] T 28029d4e66d1453cbde9811ffc2a3283 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:07.809371 7913 ts_tablet_manager.cc:1875] T 28029d4e66d1453cbde9811ffc2a3283 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:07.810163 4356 tablet_service.cc:1514] Processing DeleteTablet for tablet 47c5087093214fbeae35590b4b3c3d42 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:07 UTC) from {username='slave'} at 127.0.0.1:38306
I20240502 20:46:07.810669 7908 ts_tablet_manager.cc:1888] T 02bddc1866e04858894e047ec100a6f0 P a4bf1df7fa6b4948be502c182bfbb833: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:07.810743 7908 log.cc:1196] T 02bddc1866e04858894e047ec100a6f0 P a4bf1df7fa6b4948be502c182bfbb833: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/wal/wals/02bddc1866e04858894e047ec100a6f0
I20240502 20:46:07.811184 4354 tablet_service.cc:1514] Processing DeleteTablet for tablet 02bddc1866e04858894e047ec100a6f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:07 UTC) from {username='slave'} at 127.0.0.1:38306
I20240502 20:46:07.811389 7913 ts_tablet_manager.cc:1888] T 28029d4e66d1453cbde9811ffc2a3283 P 923cfab926e7443f8ffd7b21ad0a9d91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:07.811461 7913 log.cc:1196] T 28029d4e66d1453cbde9811ffc2a3283 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal/wals/28029d4e66d1453cbde9811ffc2a3283
I20240502 20:46:07.812043 7299 tablet_replica.cc:330] T 47c5087093214fbeae35590b4b3c3d42 P ed47c023452a497192f35d2f617bf9bd: stopping tablet replica
I20240502 20:46:07.812182 7299 raft_consensus.cc:2229] T 47c5087093214fbeae35590b4b3c3d42 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:07.812319 7299 raft_consensus.cc:2258] T 47c5087093214fbeae35590b4b3c3d42 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:07.812393 7297 ts_tablet_manager.cc:1888] T 28029d4e66d1453cbde9811ffc2a3283 P c92e6e46782849479c8a0c6d82276fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:07.812458 7297 log.cc:1196] T 28029d4e66d1453cbde9811ffc2a3283 P c92e6e46782849479c8a0c6d82276fb2: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal/wals/28029d4e66d1453cbde9811ffc2a3283
I20240502 20:46:07.812767 7297 ts_tablet_manager.cc:1909] T 28029d4e66d1453cbde9811ffc2a3283 P c92e6e46782849479c8a0c6d82276fb2: Deleting consensus metadata
I20240502 20:46:07.812836 7299 ts_tablet_manager.cc:1875] T 47c5087093214fbeae35590b4b3c3d42 P ed47c023452a497192f35d2f617bf9bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:07.813419 7297 tablet_replica.cc:330] T 47c5087093214fbeae35590b4b3c3d42 P c92e6e46782849479c8a0c6d82276fb2: stopping tablet replica
I20240502 20:46:07.813531 7297 raft_consensus.cc:2229] T 47c5087093214fbeae35590b4b3c3d42 P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:07.813753 7297 raft_consensus.cc:2258] T 47c5087093214fbeae35590b4b3c3d42 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:07.813977 4222 catalog_manager.cc:4830] TS c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827): tablet 28029d4e66d1453cbde9811ffc2a3283 (table rebalance_test_table_8 [id=e3df92c02b8746f28ae1f38db2a1630e]) successfully deleted
I20240502 20:46:07.814275 7297 ts_tablet_manager.cc:1875] T 47c5087093214fbeae35590b4b3c3d42 P c92e6e46782849479c8a0c6d82276fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:07.814926 7299 ts_tablet_manager.cc:1888] T 47c5087093214fbeae35590b4b3c3d42 P ed47c023452a497192f35d2f617bf9bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:07.814997 7299 log.cc:1196] T 47c5087093214fbeae35590b4b3c3d42 P ed47c023452a497192f35d2f617bf9bd: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal/wals/47c5087093214fbeae35590b4b3c3d42
I20240502 20:46:07.815291 7299 ts_tablet_manager.cc:1909] T 47c5087093214fbeae35590b4b3c3d42 P ed47c023452a497192f35d2f617bf9bd: Deleting consensus metadata
I20240502 20:46:07.815488 7310 raft_consensus.cc:482] T 747e7331d7ad4dd4b132672254037e1b P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:07.815572 7310 raft_consensus.cc:504] T 747e7331d7ad4dd4b132672254037e1b P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:07.815877 7299 tablet_replica.cc:330] T 02bddc1866e04858894e047ec100a6f0 P ed47c023452a497192f35d2f617bf9bd: stopping tablet replica
I20240502 20:46:07.815963 4222 catalog_manager.cc:4830] TS ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193): tablet 47c5087093214fbeae35590b4b3c3d42 (table rebalance_test_table_8 [id=e3df92c02b8746f28ae1f38db2a1630e]) successfully deleted
I20240502 20:46:07.815996 7299 raft_consensus.cc:2229] T 02bddc1866e04858894e047ec100a6f0 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:07.816042 7310 leader_election.cc:288] T 747e7331d7ad4dd4b132672254037e1b P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867), 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
I20240502 20:46:07.816069 7908 ts_tablet_manager.cc:1909] T 02bddc1866e04858894e047ec100a6f0 P a4bf1df7fa6b4948be502c182bfbb833: Deleting consensus metadata
I20240502 20:46:07.816220 7299 raft_consensus.cc:2258] T 02bddc1866e04858894e047ec100a6f0 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:07.816917 4220 catalog_manager.cc:4830] TS a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867): tablet 02bddc1866e04858894e047ec100a6f0 (table rebalance_test_table_8 [id=e3df92c02b8746f28ae1f38db2a1630e]) successfully deleted
I20240502 20:46:07.816920 6702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "747e7331d7ad4dd4b132672254037e1b" candidate_uuid: "a47e5c56f13843018ac584914a8d4707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7845c397230d49358ba2fad9db53f97c" is_pre_election: true
I20240502 20:46:07.817090 6702 raft_consensus.cc:2454] T 747e7331d7ad4dd4b132672254037e1b P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a47e5c56f13843018ac584914a8d4707 in term 0.
I20240502 20:46:07.817402 6902 leader_election.cc:302] T 747e7331d7ad4dd4b132672254037e1b P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7845c397230d49358ba2fad9db53f97c, a47e5c56f13843018ac584914a8d4707; no voters:
I20240502 20:46:07.817498 7297 ts_tablet_manager.cc:1888] T 47c5087093214fbeae35590b4b3c3d42 P c92e6e46782849479c8a0c6d82276fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:07.817565 7297 log.cc:1196] T 47c5087093214fbeae35590b4b3c3d42 P c92e6e46782849479c8a0c6d82276fb2: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal/wals/47c5087093214fbeae35590b4b3c3d42
I20240502 20:46:07.817588 7310 raft_consensus.cc:2781] T 747e7331d7ad4dd4b132672254037e1b P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:07.817636 7310 raft_consensus.cc:482] T 747e7331d7ad4dd4b132672254037e1b P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:07.817677 7310 raft_consensus.cc:3037] T 747e7331d7ad4dd4b132672254037e1b P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:07.817766 7297 ts_tablet_manager.cc:1909] T 47c5087093214fbeae35590b4b3c3d42 P c92e6e46782849479c8a0c6d82276fb2: Deleting consensus metadata
I20240502 20:46:07.818204 7297 tablet_replica.cc:330] T 02bddc1866e04858894e047ec100a6f0 P c92e6e46782849479c8a0c6d82276fb2: stopping tablet replica
I20240502 20:46:07.818311 4222 catalog_manager.cc:4830] TS c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827): tablet 47c5087093214fbeae35590b4b3c3d42 (table rebalance_test_table_8 [id=e3df92c02b8746f28ae1f38db2a1630e]) successfully deleted
I20240502 20:46:07.818338 7297 raft_consensus.cc:2229] T 02bddc1866e04858894e047ec100a6f0 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:07.818462 7297 raft_consensus.cc:2258] T 02bddc1866e04858894e047ec100a6f0 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:07.818588 7310 raft_consensus.cc:504] T 747e7331d7ad4dd4b132672254037e1b P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:07.818850 7297 ts_tablet_manager.cc:1875] T 02bddc1866e04858894e047ec100a6f0 P c92e6e46782849479c8a0c6d82276fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:07.818903 7310 leader_election.cc:288] T 747e7331d7ad4dd4b132672254037e1b P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 1 election: Requested vote from peers a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867), 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
I20240502 20:46:07.819363 6702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "747e7331d7ad4dd4b132672254037e1b" candidate_uuid: "a47e5c56f13843018ac584914a8d4707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7845c397230d49358ba2fad9db53f97c"
I20240502 20:46:07.819473 6702 raft_consensus.cc:3037] T 747e7331d7ad4dd4b132672254037e1b P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:07.820300 7297 ts_tablet_manager.cc:1888] T 02bddc1866e04858894e047ec100a6f0 P c92e6e46782849479c8a0c6d82276fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:07.820365 7297 log.cc:1196] T 02bddc1866e04858894e047ec100a6f0 P c92e6e46782849479c8a0c6d82276fb2: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal/wals/02bddc1866e04858894e047ec100a6f0
I20240502 20:46:07.820365 6702 raft_consensus.cc:2454] T 747e7331d7ad4dd4b132672254037e1b P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a47e5c56f13843018ac584914a8d4707 in term 1.
I20240502 20:46:07.820581 7297 ts_tablet_manager.cc:1909] T 02bddc1866e04858894e047ec100a6f0 P c92e6e46782849479c8a0c6d82276fb2: Deleting consensus metadata
I20240502 20:46:07.820637 6902 leader_election.cc:302] T 747e7331d7ad4dd4b132672254037e1b P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7845c397230d49358ba2fad9db53f97c, a47e5c56f13843018ac584914a8d4707; no voters:
I20240502 20:46:07.820793 7310 raft_consensus.cc:2781] T 747e7331d7ad4dd4b132672254037e1b P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:07.820869 7310 raft_consensus.cc:686] T 747e7331d7ad4dd4b132672254037e1b P a47e5c56f13843018ac584914a8d4707 [term 1 LEADER]: Becoming Leader. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Running, Role: LEADER
I20240502 20:46:07.820956 8138 tablet_replica.cc:330] T 47c5087093214fbeae35590b4b3c3d42 P a47e5c56f13843018ac584914a8d4707: stopping tablet replica
I20240502 20:46:07.821101 8138 raft_consensus.cc:2229] T 47c5087093214fbeae35590b4b3c3d42 P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Raft consensus shutting down.
I20240502 20:46:07.821192 8138 raft_consensus.cc:2258] T 47c5087093214fbeae35590b4b3c3d42 P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Raft consensus is shut down!
I20240502 20:46:07.821328 4222 catalog_manager.cc:4830] TS c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827): tablet 02bddc1866e04858894e047ec100a6f0 (table rebalance_test_table_8 [id=e3df92c02b8746f28ae1f38db2a1630e]) successfully deleted
I20240502 20:46:07.821324 7310 consensus_queue.cc:237] T 747e7331d7ad4dd4b132672254037e1b P a47e5c56f13843018ac584914a8d4707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:07.821638 8138 ts_tablet_manager.cc:1875] T 47c5087093214fbeae35590b4b3c3d42 P a47e5c56f13843018ac584914a8d4707: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:07.822234 4618 tablet_service.cc:1514] Processing DeleteTablet for tablet 02bddc1866e04858894e047ec100a6f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:07 UTC) from {username='slave'} at 127.0.0.1:57068
I20240502 20:46:07.823369 8137 tablet_replica.cc:330] T 28029d4e66d1453cbde9811ffc2a3283 P 201d2b6485084e7dad2cbb722af94779: stopping tablet replica
I20240502 20:46:07.823725 8137 raft_consensus.cc:2229] T 28029d4e66d1453cbde9811ffc2a3283 P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Raft consensus shutting down.
I20240502 20:46:07.824142 4488 tablet_service.cc:1514] Processing DeleteTablet for tablet 28029d4e66d1453cbde9811ffc2a3283 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:07 UTC) from {username='slave'} at 127.0.0.1:48890
I20240502 20:46:07.824266 4487 tablet_service.cc:1514] Processing DeleteTablet for tablet 47c5087093214fbeae35590b4b3c3d42 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:07 UTC) from {username='slave'} at 127.0.0.1:48890
I20240502 20:46:07.824323 4231 catalog_manager.cc:5484] T 747e7331d7ad4dd4b132672254037e1b P a47e5c56f13843018ac584914a8d4707 reported cstate change: term changed from 0 to 1, leader changed from <none> to a47e5c56f13843018ac584914a8d4707 (127.30.32.71). New cstate: current_term: 1 leader_uuid: "a47e5c56f13843018ac584914a8d4707" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: HEALTHY } } }
I20240502 20:46:07.824622 7921 raft_consensus.cc:1050] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc: attempting to promote NON_VOTER 7845c397230d49358ba2fad9db53f97c to VOTER
I20240502 20:46:07.824642 7299 ts_tablet_manager.cc:1875] T 02bddc1866e04858894e047ec100a6f0 P ed47c023452a497192f35d2f617bf9bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:07.824864 8137 raft_consensus.cc:2258] T 28029d4e66d1453cbde9811ffc2a3283 P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Raft consensus is shut down!
I20240502 20:46:07.825054 7921 consensus_queue.cc:237] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } }
I20240502 20:46:07.825876 8137 ts_tablet_manager.cc:1875] T 28029d4e66d1453cbde9811ffc2a3283 P 201d2b6485084e7dad2cbb722af94779: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:07.826013 7913 ts_tablet_manager.cc:1909] T 28029d4e66d1453cbde9811ffc2a3283 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting consensus metadata
I20240502 20:46:07.826639 7913 tablet_replica.cc:330] T 02bddc1866e04858894e047ec100a6f0 P 923cfab926e7443f8ffd7b21ad0a9d91: stopping tablet replica
I20240502 20:46:07.826776 7913 raft_consensus.cc:2229] T 02bddc1866e04858894e047ec100a6f0 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:07.826813 6703 raft_consensus.cc:1261] T a62a7ce526f14c25994bb9e0e3c3bc84 P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:07.826905 7913 raft_consensus.cc:2258] T 02bddc1866e04858894e047ec100a6f0 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:07.827319 7404 consensus_queue.cc:1035] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:07.827450 7913 ts_tablet_manager.cc:1875] T 02bddc1866e04858894e047ec100a6f0 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:07.827507 4219 catalog_manager.cc:4830] TS 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141): tablet 28029d4e66d1453cbde9811ffc2a3283 (table rebalance_test_table_8 [id=e3df92c02b8746f28ae1f38db2a1630e]) successfully deleted
I20240502 20:46:07.827947 8138 ts_tablet_manager.cc:1888] T 47c5087093214fbeae35590b4b3c3d42 P a47e5c56f13843018ac584914a8d4707: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:07.828011 8138 log.cc:1196] T 47c5087093214fbeae35590b4b3c3d42 P a47e5c56f13843018ac584914a8d4707: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/wal/wals/47c5087093214fbeae35590b4b3c3d42
I20240502 20:46:07.828298 8138 ts_tablet_manager.cc:1909] T 47c5087093214fbeae35590b4b3c3d42 P a47e5c56f13843018ac584914a8d4707: Deleting consensus metadata
I20240502 20:46:07.828878 4374 raft_consensus.cc:1261] T a62a7ce526f14c25994bb9e0e3c3bc84 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:07.829959 8137 ts_tablet_manager.cc:1888] T 28029d4e66d1453cbde9811ffc2a3283 P 201d2b6485084e7dad2cbb722af94779: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:07.830090 7404 consensus_queue.cc:1035] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:07.830391 8137 log.cc:1196] T 28029d4e66d1453cbde9811ffc2a3283 P 201d2b6485084e7dad2cbb722af94779: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/wal/wals/28029d4e66d1453cbde9811ffc2a3283
I20240502 20:46:07.830508 4222 catalog_manager.cc:4830] TS a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): tablet 47c5087093214fbeae35590b4b3c3d42 (table rebalance_test_table_8 [id=e3df92c02b8746f28ae1f38db2a1630e]) successfully deleted
I20240502 20:46:07.830868 8137 ts_tablet_manager.cc:1909] T 28029d4e66d1453cbde9811ffc2a3283 P 201d2b6485084e7dad2cbb722af94779: Deleting consensus metadata
I20240502 20:46:07.830960 4637 raft_consensus.cc:1261] T a62a7ce526f14c25994bb9e0e3c3bc84 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:07.831055 7299 ts_tablet_manager.cc:1888] T 02bddc1866e04858894e047ec100a6f0 P ed47c023452a497192f35d2f617bf9bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:07.831140 7299 log.cc:1196] T 02bddc1866e04858894e047ec100a6f0 P ed47c023452a497192f35d2f617bf9bd: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal/wals/02bddc1866e04858894e047ec100a6f0
I20240502 20:46:07.831434 7299 ts_tablet_manager.cc:1909] T 02bddc1866e04858894e047ec100a6f0 P ed47c023452a497192f35d2f617bf9bd: Deleting consensus metadata
I20240502 20:46:07.832031 7404 consensus_queue.cc:1035] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:07.834875 4219 catalog_manager.cc:4830] TS 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): tablet 28029d4e66d1453cbde9811ffc2a3283 (table rebalance_test_table_8 [id=e3df92c02b8746f28ae1f38db2a1630e]) successfully deleted
I20240502 20:46:07.834848 4508 consensus_queue.cc:237] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } }
I20240502 20:46:07.835253 8141 tablet_replica.cc:330] T 28029d4e66d1453cbde9811ffc2a3283 P c3d26eafd51547acb780d4aac5ebe7cc: stopping tablet replica
I20240502 20:46:07.835433 8141 raft_consensus.cc:2229] T 28029d4e66d1453cbde9811ffc2a3283 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:07.835701 8141 raft_consensus.cc:2258] T 28029d4e66d1453cbde9811ffc2a3283 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:07.836337 8141 ts_tablet_manager.cc:1875] T 28029d4e66d1453cbde9811ffc2a3283 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:07.837167 4768 raft_consensus.cc:1261] T d6fb394fdcff436886fe0634d79c024c P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:07.837932 6702 raft_consensus.cc:1261] T d6fb394fdcff436886fe0634d79c024c P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:07.838220 7913 ts_tablet_manager.cc:1888] T 02bddc1866e04858894e047ec100a6f0 P 923cfab926e7443f8ffd7b21ad0a9d91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:07.838356 7404 consensus_queue.cc:1035] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:07.839104 7913 log.cc:1196] T 02bddc1866e04858894e047ec100a6f0 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal/wals/02bddc1866e04858894e047ec100a6f0
I20240502 20:46:07.839150 8141 ts_tablet_manager.cc:1888] T 28029d4e66d1453cbde9811ffc2a3283 P c3d26eafd51547acb780d4aac5ebe7cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:07.839274 8141 log.cc:1196] T 28029d4e66d1453cbde9811ffc2a3283 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal/wals/28029d4e66d1453cbde9811ffc2a3283
I20240502 20:46:07.839593 8141 ts_tablet_manager.cc:1909] T 28029d4e66d1453cbde9811ffc2a3283 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting consensus metadata
I20240502 20:46:07.839833 7913 ts_tablet_manager.cc:1909] T 02bddc1866e04858894e047ec100a6f0 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting consensus metadata
I20240502 20:46:07.840561 8141 tablet_replica.cc:330] T 47c5087093214fbeae35590b4b3c3d42 P c3d26eafd51547acb780d4aac5ebe7cc: stopping tablet replica
I20240502 20:46:07.840726 8141 raft_consensus.cc:2229] T 47c5087093214fbeae35590b4b3c3d42 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:07.840765 7404 raft_consensus.cc:2932] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:07.841339 4221 catalog_manager.cc:4830] TS c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097): tablet 28029d4e66d1453cbde9811ffc2a3283 (table rebalance_test_table_8 [id=e3df92c02b8746f28ae1f38db2a1630e]) successfully deleted
I20240502 20:46:07.840919 8141 raft_consensus.cc:2258] T 47c5087093214fbeae35590b4b3c3d42 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:07.841682 7921 consensus_queue.cc:1035] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:07.841885 4222 catalog_manager.cc:4830] TS ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193): tablet 02bddc1866e04858894e047ec100a6f0 (table rebalance_test_table_8 [id=e3df92c02b8746f28ae1f38db2a1630e]) successfully deleted
I20240502 20:46:07.842171 7921 raft_consensus.cc:2932] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:07.842491 4219 catalog_manager.cc:4830] TS 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141): tablet 02bddc1866e04858894e047ec100a6f0 (table rebalance_test_table_8 [id=e3df92c02b8746f28ae1f38db2a1630e]) successfully deleted
I20240502 20:46:07.843010 6703 raft_consensus.cc:2932] T a62a7ce526f14c25994bb9e0e3c3bc84 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:07.845415 6702 raft_consensus.cc:2932] T d6fb394fdcff436886fe0634d79c024c P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:07.845793 4221 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d6fb394fdcff436886fe0634d79c024c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:07.846220 4374 raft_consensus.cc:2932] T a62a7ce526f14c25994bb9e0e3c3bc84 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:07.847273 8141 ts_tablet_manager.cc:1875] T 47c5087093214fbeae35590b4b3c3d42 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:07.847658 4637 raft_consensus.cc:2932] T a62a7ce526f14c25994bb9e0e3c3bc84 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:07.849574 8141 ts_tablet_manager.cc:1888] T 47c5087093214fbeae35590b4b3c3d42 P c3d26eafd51547acb780d4aac5ebe7cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:07.849690 8141 log.cc:1196] T 47c5087093214fbeae35590b4b3c3d42 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal/wals/47c5087093214fbeae35590b4b3c3d42
I20240502 20:46:07.850019 8141 ts_tablet_manager.cc:1909] T 47c5087093214fbeae35590b4b3c3d42 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting consensus metadata
I20240502 20:46:07.850020 4233 catalog_manager.cc:5484] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: config changed from index 4 to 5, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:07.850353 4233 catalog_manager.cc:5484] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: config changed from index 5 to 6, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:07.850812 4768 raft_consensus.cc:2932] T d6fb394fdcff436886fe0634d79c024c P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:07.853232 6835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "747e7331d7ad4dd4b132672254037e1b" candidate_uuid: "a47e5c56f13843018ac584914a8d4707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833" is_pre_election: true
I20240502 20:46:07.853370 6835 raft_consensus.cc:2454] T 747e7331d7ad4dd4b132672254037e1b P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a47e5c56f13843018ac584914a8d4707 in term 0.
I20240502 20:46:07.853554 6831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "747e7331d7ad4dd4b132672254037e1b" candidate_uuid: "a47e5c56f13843018ac584914a8d4707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833"
I20240502 20:46:07.853646 6831 raft_consensus.cc:3037] T 747e7331d7ad4dd4b132672254037e1b P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:07.854599 6831 raft_consensus.cc:2454] T 747e7331d7ad4dd4b132672254037e1b P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a47e5c56f13843018ac584914a8d4707 in term 1.
I20240502 20:46:07.856851 4618 tablet_service.cc:1514] Processing DeleteTablet for tablet d6fb394fdcff436886fe0634d79c024c with delete_type TABLET_DATA_TOMBSTONED (TS 923cfab926e7443f8ffd7b21ad0a9d91 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57068
I20240502 20:46:07.857645 4508 consensus_queue.cc:237] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } }
I20240502 20:46:07.858062 7913 tablet_replica.cc:330] T d6fb394fdcff436886fe0634d79c024c P 923cfab926e7443f8ffd7b21ad0a9d91: stopping tablet replica
I20240502 20:46:07.858266 7913 raft_consensus.cc:2229] T d6fb394fdcff436886fe0634d79c024c P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:07.858502 7913 raft_consensus.cc:2258] T d6fb394fdcff436886fe0634d79c024c P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:07.858762 6702 raft_consensus.cc:1261] T a62a7ce526f14c25994bb9e0e3c3bc84 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:07.858923 4374 raft_consensus.cc:1261] T a62a7ce526f14c25994bb9e0e3c3bc84 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:07.859194 7921 consensus_queue.cc:1035] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:07.859817 7921 consensus_queue.cc:1035] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:07.860442 7913 ts_tablet_manager.cc:1875] T d6fb394fdcff436886fe0634d79c024c P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:07.862293 4221 catalog_manager.cc:4830] TS c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097): tablet 47c5087093214fbeae35590b4b3c3d42 (table rebalance_test_table_8 [id=e3df92c02b8746f28ae1f38db2a1630e]) successfully deleted
I20240502 20:46:07.862465 7996 raft_consensus.cc:1050] T f352efa74d3c4da3907efdc37237211f P ed47c023452a497192f35d2f617bf9bd: attempting to promote NON_VOTER 201d2b6485084e7dad2cbb722af94779 to VOTER
I20240502 20:46:07.862907 7996 consensus_queue.cc:237] T f352efa74d3c4da3907efdc37237211f P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } }
I20240502 20:46:07.863312 7913 ts_tablet_manager.cc:1888] T d6fb394fdcff436886fe0634d79c024c P 923cfab926e7443f8ffd7b21ad0a9d91: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:46:07.863386 7913 log.cc:1196] T d6fb394fdcff436886fe0634d79c024c P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal/wals/d6fb394fdcff436886fe0634d79c024c
I20240502 20:46:07.864499 4508 raft_consensus.cc:1261] T f352efa74d3c4da3907efdc37237211f P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:07.865051 7098 raft_consensus.cc:1261] T f352efa74d3c4da3907efdc37237211f P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:07.864903 7921 raft_consensus.cc:2932] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:07.865322 4219 catalog_manager.cc:4830] TS 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141): tablet d6fb394fdcff436886fe0634d79c024c (table rebalance_test_table_9 [id=7c90eeb05db84afb8f78ab7f14e0ced4]) successfully deleted
I20240502 20:46:07.865470 6241 consensus_queue.cc:1035] T f352efa74d3c4da3907efdc37237211f P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:07.865729 4374 raft_consensus.cc:2932] T a62a7ce526f14c25994bb9e0e3c3bc84 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:07.866593 6241 consensus_queue.cc:1035] T f352efa74d3c4da3907efdc37237211f P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:07.866717 4221 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a62a7ce526f14c25994bb9e0e3c3bc84 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:07.867977 6702 raft_consensus.cc:2932] T a62a7ce526f14c25994bb9e0e3c3bc84 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:07.870033 6241 raft_consensus.cc:1050] T 9e34e8e17a7e413b97eb614a7df727d5 P ed47c023452a497192f35d2f617bf9bd: attempting to promote NON_VOTER 201d2b6485084e7dad2cbb722af94779 to VOTER
I20240502 20:46:07.870448 6241 consensus_queue.cc:237] T 9e34e8e17a7e413b97eb614a7df727d5 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } }
I20240502 20:46:07.870975 4227 catalog_manager.cc:5484] T a62a7ce526f14c25994bb9e0e3c3bc84 P 7845c397230d49358ba2fad9db53f97c reported cstate change: config changed from index 5 to 6, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:07.872834 7099 raft_consensus.cc:1261] T 9e34e8e17a7e413b97eb614a7df727d5 P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:07.873688 4618 tablet_service.cc:1514] Processing DeleteTablet for tablet a62a7ce526f14c25994bb9e0e3c3bc84 with delete_type TABLET_DATA_TOMBSTONED (TS 923cfab926e7443f8ffd7b21ad0a9d91 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57068
I20240502 20:46:07.873956 7913 tablet_replica.cc:330] T a62a7ce526f14c25994bb9e0e3c3bc84 P 923cfab926e7443f8ffd7b21ad0a9d91: stopping tablet replica
I20240502 20:46:07.874040 4506 raft_consensus.cc:1261] T 9e34e8e17a7e413b97eb614a7df727d5 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:07.874040 8111 consensus_queue.cc:1035] T 9e34e8e17a7e413b97eb614a7df727d5 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:07.874192 7913 raft_consensus.cc:2229] T a62a7ce526f14c25994bb9e0e3c3bc84 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:07.874415 7913 raft_consensus.cc:2258] T a62a7ce526f14c25994bb9e0e3c3bc84 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:07.874907 8111 consensus_queue.cc:1035] T 9e34e8e17a7e413b97eb614a7df727d5 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:07.875314 7913 ts_tablet_manager.cc:1875] T a62a7ce526f14c25994bb9e0e3c3bc84 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:07.876147 4637 raft_consensus.cc:1261] T f352efa74d3c4da3907efdc37237211f P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:07.876559 7318 consensus_queue.cc:1035] T f352efa74d3c4da3907efdc37237211f P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:07.877817 7913 ts_tablet_manager.cc:1888] T a62a7ce526f14c25994bb9e0e3c3bc84 P 923cfab926e7443f8ffd7b21ad0a9d91: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:46:07.877938 7913 log.cc:1196] T a62a7ce526f14c25994bb9e0e3c3bc84 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal/wals/a62a7ce526f14c25994bb9e0e3c3bc84
I20240502 20:46:07.878213 4374 raft_consensus.cc:1261] T 9e34e8e17a7e413b97eb614a7df727d5 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:07.878552 4219 catalog_manager.cc:4830] TS 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141): tablet a62a7ce526f14c25994bb9e0e3c3bc84 (table rebalance_test_table_17 [id=75dfa602f9634de99d6f55e4fdeea21c]) successfully deleted
I20240502 20:46:07.878623 7996 consensus_queue.cc:1035] T 9e34e8e17a7e413b97eb614a7df727d5 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:07.888842 4228 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 20:46:07.889940 7996 raft_consensus.cc:2932] T 9e34e8e17a7e413b97eb614a7df727d5 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:07.891352 7079 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:46:07.892455 4374 raft_consensus.cc:2932] T 9e34e8e17a7e413b97eb614a7df727d5 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:07.892668 4506 raft_consensus.cc:2932] T 9e34e8e17a7e413b97eb614a7df727d5 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:07.894480 7098 raft_consensus.cc:2932] T 9e34e8e17a7e413b97eb614a7df727d5 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:07.895463 4233 catalog_manager.cc:5484] T 9e34e8e17a7e413b97eb614a7df727d5 P c92e6e46782849479c8a0c6d82276fb2 reported cstate change: config changed from index 4 to 5, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:07.898531 8027 tablet.cc:1722] T a25f2855fb7d4eac9f1c920fdd01e2ba P 201d2b6485084e7dad2cbb722af94779: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 20:46:07.900429 8035 tablet.cc:1722] T a25f2855fb7d4eac9f1c920fdd01e2ba P a47e5c56f13843018ac584914a8d4707: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 20:46:07.901268 4768 consensus_queue.cc:237] T 9e34e8e17a7e413b97eb614a7df727d5 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } }
I20240502 20:46:07.902514 4377 raft_consensus.cc:1261] T 9e34e8e17a7e413b97eb614a7df727d5 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:07.903386 6241 consensus_queue.cc:1035] T 9e34e8e17a7e413b97eb614a7df727d5 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:07.906709 7098 raft_consensus.cc:1261] T 9e34e8e17a7e413b97eb614a7df727d5 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:07.907199 8111 consensus_queue.cc:1035] T 9e34e8e17a7e413b97eb614a7df727d5 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:07.907568 8029 tablet.cc:1722] T a25f2855fb7d4eac9f1c920fdd01e2ba P c92e6e46782849479c8a0c6d82276fb2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 20:46:07.907948 4234 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: rebalancer_dynamic_table_new_name [id=0d23318bd0f5482face4d225e00e7757] alter complete (version 3)
I20240502 20:46:07.912035 8111 raft_consensus.cc:2932] T 9e34e8e17a7e413b97eb614a7df727d5 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:07.912492 7098 raft_consensus.cc:2932] T 9e34e8e17a7e413b97eb614a7df727d5 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:07.914124 4222 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9e34e8e17a7e413b97eb614a7df727d5 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:07.914034 4374 raft_consensus.cc:2932] T 9e34e8e17a7e413b97eb614a7df727d5 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:07.915274 4228 catalog_manager.cc:5484] T 9e34e8e17a7e413b97eb614a7df727d5 P 201d2b6485084e7dad2cbb722af94779 reported cstate change: config changed from index 5 to 6, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:07.916000 7996 raft_consensus.cc:2932] T f352efa74d3c4da3907efdc37237211f P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:07.918342 4487 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e34e8e17a7e413b97eb614a7df727d5 with delete_type TABLET_DATA_TOMBSTONED (TS c3d26eafd51547acb780d4aac5ebe7cc not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48890
I20240502 20:46:07.918586 8141 tablet_replica.cc:330] T 9e34e8e17a7e413b97eb614a7df727d5 P c3d26eafd51547acb780d4aac5ebe7cc: stopping tablet replica
I20240502 20:46:07.918745 8141 raft_consensus.cc:2229] T 9e34e8e17a7e413b97eb614a7df727d5 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:07.918888 8141 raft_consensus.cc:2258] T 9e34e8e17a7e413b97eb614a7df727d5 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:07.919317 7098 raft_consensus.cc:2932] T f352efa74d3c4da3907efdc37237211f P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:07.919548 8141 ts_tablet_manager.cc:1875] T 9e34e8e17a7e413b97eb614a7df727d5 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:07.919436 4637 raft_consensus.cc:2932] T f352efa74d3c4da3907efdc37237211f P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:07.922348 4228 catalog_manager.cc:5484] T f352efa74d3c4da3907efdc37237211f P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index 4 to 5, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:07.922554 8141 ts_tablet_manager.cc:1888] T 9e34e8e17a7e413b97eb614a7df727d5 P c3d26eafd51547acb780d4aac5ebe7cc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:46:07.922622 8141 log.cc:1196] T 9e34e8e17a7e413b97eb614a7df727d5 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal/wals/9e34e8e17a7e413b97eb614a7df727d5
I20240502 20:46:07.922752 4508 raft_consensus.cc:2932] T f352efa74d3c4da3907efdc37237211f P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:07.923947 4221 catalog_manager.cc:4830] TS c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097): tablet 9e34e8e17a7e413b97eb614a7df727d5 (table rebalance_test_table_10 [id=e54c8dd95147474b9f91de67bc9c075c]) successfully deleted
I20240502 20:46:07.925908 4768 consensus_queue.cc:237] T f352efa74d3c4da3907efdc37237211f P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } }
I20240502 20:46:07.927124 4637 raft_consensus.cc:1261] T f352efa74d3c4da3907efdc37237211f P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:07.927219 7098 raft_consensus.cc:1261] T f352efa74d3c4da3907efdc37237211f P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:07.927738 6241 consensus_queue.cc:1035] T f352efa74d3c4da3907efdc37237211f P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:07.928045 6241 consensus_queue.cc:1035] T f352efa74d3c4da3907efdc37237211f P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:07.929982 7318 raft_consensus.cc:2932] T f352efa74d3c4da3907efdc37237211f P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:07.931859 4222 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f352efa74d3c4da3907efdc37237211f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:07.932147 4637 raft_consensus.cc:2932] T f352efa74d3c4da3907efdc37237211f P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:07.934639 4234 catalog_manager.cc:5484] T f352efa74d3c4da3907efdc37237211f P ed47c023452a497192f35d2f617bf9bd reported cstate change: config changed from index 5 to 6, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:07.934885 7099 raft_consensus.cc:2932] T f352efa74d3c4da3907efdc37237211f P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:07.938429 4487 tablet_service.cc:1514] Processing DeleteTablet for tablet f352efa74d3c4da3907efdc37237211f with delete_type TABLET_DATA_TOMBSTONED (TS c3d26eafd51547acb780d4aac5ebe7cc not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48890
I20240502 20:46:07.938685 8141 tablet_replica.cc:330] T f352efa74d3c4da3907efdc37237211f P c3d26eafd51547acb780d4aac5ebe7cc: stopping tablet replica
I20240502 20:46:07.938874 8141 raft_consensus.cc:2229] T f352efa74d3c4da3907efdc37237211f P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:07.939095 8141 raft_consensus.cc:2258] T f352efa74d3c4da3907efdc37237211f P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:07.939867 8141 ts_tablet_manager.cc:1875] T f352efa74d3c4da3907efdc37237211f P c3d26eafd51547acb780d4aac5ebe7cc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:07.941839 6683 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:07.942644 8141 ts_tablet_manager.cc:1888] T f352efa74d3c4da3907efdc37237211f P c3d26eafd51547acb780d4aac5ebe7cc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:46:07.942724 8141 log.cc:1196] T f352efa74d3c4da3907efdc37237211f P c3d26eafd51547acb780d4aac5ebe7cc: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal/wals/f352efa74d3c4da3907efdc37237211f
I20240502 20:46:07.943281 4221 catalog_manager.cc:4830] TS c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097): tablet f352efa74d3c4da3907efdc37237211f (table rebalance_test_table_12 [id=dab4c58041724066a7c74bff06307285]) successfully deleted
I20240502 20:46:07.959439 4487 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 20:46:07.961021 6947 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:07.963270 4354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:46:07.964444 7017 raft_consensus.cc:1050] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91: attempting to promote NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 to VOTER
I20240502 20:46:07.965118 4750 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 20:46:07.965024 7017 consensus_queue.cc:237] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } }
I20240502 20:46:07.965206 4618 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 20:46:07.967007 6831 raft_consensus.cc:1261] T 7abfd61f55554042978b50761cbd6ab5 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEARNER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:07.967274 4374 raft_consensus.cc:1261] T 7abfd61f55554042978b50761cbd6ab5 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:07.967461 8168 consensus_queue.cc:1035] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:07.967604 4768 raft_consensus.cc:1261] T 7abfd61f55554042978b50761cbd6ab5 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:07.968165 6613 consensus_queue.cc:1035] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:07.968478 7447 consensus_queue.cc:1035] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:07.973496 6809 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:07.995005 8168 raft_consensus.cc:2932] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:07.997184 6831 raft_consensus.cc:2932] T 7abfd61f55554042978b50761cbd6ab5 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:07.998013 4228 catalog_manager.cc:5484] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index 4 to 5, a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:07.998068 4374 raft_consensus.cc:2932] T 7abfd61f55554042978b50761cbd6ab5 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:08.005795 4768 raft_consensus.cc:2932] T 7abfd61f55554042978b50761cbd6ab5 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:08.010185 4637 consensus_queue.cc:237] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } }
I20240502 20:46:08.011507 4234 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 20:46:08.011638 4234 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 20:46:08.012929 6831 raft_consensus.cc:1261] T 7abfd61f55554042978b50761cbd6ab5 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:08.013350 6613 consensus_queue.cc:1035] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:08.013414 4768 raft_consensus.cc:1261] T 7abfd61f55554042978b50761cbd6ab5 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:08.013801 6613 consensus_queue.cc:1035] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:08.014384 4234 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: Sending DeleteTablet for 3 replicas of tablet a25f2855fb7d4eac9f1c920fdd01e2ba
I20240502 20:46:08.014940 7079 tablet_service.cc:1514] Processing DeleteTablet for tablet a25f2855fb7d4eac9f1c920fdd01e2ba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:08 UTC) from {username='slave'} at 127.0.0.1:59162
I20240502 20:46:08.015185 8137 tablet_replica.cc:330] T a25f2855fb7d4eac9f1c920fdd01e2ba P 201d2b6485084e7dad2cbb722af94779: stopping tablet replica
I20240502 20:46:08.015264 4354 tablet_service.cc:1514] Processing DeleteTablet for tablet a25f2855fb7d4eac9f1c920fdd01e2ba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:08 UTC) from {username='slave'} at 127.0.0.1:38306
I20240502 20:46:08.015331 8137 raft_consensus.cc:2229] T a25f2855fb7d4eac9f1c920fdd01e2ba P 201d2b6485084e7dad2cbb722af94779 [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:08.015470 7297 tablet_replica.cc:330] T a25f2855fb7d4eac9f1c920fdd01e2ba P c92e6e46782849479c8a0c6d82276fb2: stopping tablet replica
I20240502 20:46:08.015619 7297 raft_consensus.cc:2229] T a25f2855fb7d4eac9f1c920fdd01e2ba P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:08.015671 8137 raft_consensus.cc:2258] T a25f2855fb7d4eac9f1c920fdd01e2ba P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:08.015736 7297 raft_consensus.cc:2258] T a25f2855fb7d4eac9f1c920fdd01e2ba P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:08.015909 6947 tablet_service.cc:1514] Processing DeleteTablet for tablet a25f2855fb7d4eac9f1c920fdd01e2ba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:08 UTC) from {username='slave'} at 127.0.0.1:45156
I20240502 20:46:08.016238 8138 tablet_replica.cc:330] T a25f2855fb7d4eac9f1c920fdd01e2ba P a47e5c56f13843018ac584914a8d4707: stopping tablet replica
I20240502 20:46:08.016355 8138 raft_consensus.cc:2229] T a25f2855fb7d4eac9f1c920fdd01e2ba P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:08.016350 7297 ts_tablet_manager.cc:1875] T a25f2855fb7d4eac9f1c920fdd01e2ba P c92e6e46782849479c8a0c6d82276fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:08.016459 8138 raft_consensus.cc:2258] T a25f2855fb7d4eac9f1c920fdd01e2ba P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:08.017642 8138 ts_tablet_manager.cc:1875] T a25f2855fb7d4eac9f1c920fdd01e2ba P a47e5c56f13843018ac584914a8d4707: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:08.018663 7297 ts_tablet_manager.cc:1888] T a25f2855fb7d4eac9f1c920fdd01e2ba P c92e6e46782849479c8a0c6d82276fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:08.018728 7297 log.cc:1196] T a25f2855fb7d4eac9f1c920fdd01e2ba P c92e6e46782849479c8a0c6d82276fb2: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal/wals/a25f2855fb7d4eac9f1c920fdd01e2ba
I20240502 20:46:08.018998 7297 ts_tablet_manager.cc:1909] T a25f2855fb7d4eac9f1c920fdd01e2ba P c92e6e46782849479c8a0c6d82276fb2: Deleting consensus metadata
I20240502 20:46:08.019044 6613 raft_consensus.cc:2932] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:08.019672 8138 ts_tablet_manager.cc:1888] T a25f2855fb7d4eac9f1c920fdd01e2ba P a47e5c56f13843018ac584914a8d4707: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:08.019731 8138 log.cc:1196] T a25f2855fb7d4eac9f1c920fdd01e2ba P a47e5c56f13843018ac584914a8d4707: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/wal/wals/a25f2855fb7d4eac9f1c920fdd01e2ba
I20240502 20:46:08.019984 8138 ts_tablet_manager.cc:1909] T a25f2855fb7d4eac9f1c920fdd01e2ba P a47e5c56f13843018ac584914a8d4707: Deleting consensus metadata
I20240502 20:46:08.020916 4222 catalog_manager.cc:4830] TS c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827): tablet a25f2855fb7d4eac9f1c920fdd01e2ba (table rebalancer_dynamic_table_new_name [id=0d23318bd0f5482face4d225e00e7757]) successfully deleted
I20240502 20:46:08.021075 4219 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7abfd61f55554042978b50761cbd6ab5 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:08.021582 6831 raft_consensus.cc:2932] T 7abfd61f55554042978b50761cbd6ab5 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:08.021767 8137 ts_tablet_manager.cc:1875] T a25f2855fb7d4eac9f1c920fdd01e2ba P 201d2b6485084e7dad2cbb722af94779: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:08.021853 8168 raft_consensus.cc:1050] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91: attempting to promote NON_VOTER 7845c397230d49358ba2fad9db53f97c to VOTER
I20240502 20:46:08.022229 8168 consensus_queue.cc:237] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } }
I20240502 20:46:08.023680 4506 raft_consensus.cc:1261] T 00eb91e624554baf918e96b098019be8 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:08.023849 4222 catalog_manager.cc:4830] TS a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): tablet a25f2855fb7d4eac9f1c920fdd01e2ba (table rebalancer_dynamic_table_new_name [id=0d23318bd0f5482face4d225e00e7757]) successfully deleted
I20240502 20:46:08.023922 8137 ts_tablet_manager.cc:1888] T a25f2855fb7d4eac9f1c920fdd01e2ba P 201d2b6485084e7dad2cbb722af94779: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:08.023985 8137 log.cc:1196] T a25f2855fb7d4eac9f1c920fdd01e2ba P 201d2b6485084e7dad2cbb722af94779: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/wal/wals/a25f2855fb7d4eac9f1c920fdd01e2ba
I20240502 20:46:08.024195 7017 consensus_queue.cc:1035] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:08.024258 8137 ts_tablet_manager.cc:1909] T a25f2855fb7d4eac9f1c920fdd01e2ba P 201d2b6485084e7dad2cbb722af94779: Deleting consensus metadata
I20240502 20:46:08.024560 4767 raft_consensus.cc:1261] T 00eb91e624554baf918e96b098019be8 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:08.024961 6702 raft_consensus.cc:1261] T 00eb91e624554baf918e96b098019be8 P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:08.025100 7017 consensus_queue.cc:1035] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:08.025264 4229 catalog_manager.cc:5484] T 7abfd61f55554042978b50761cbd6ab5 P a4bf1df7fa6b4948be502c182bfbb833 reported cstate change: config changed from index 5 to 6, VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) evicted. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:08.027420 4219 catalog_manager.cc:4830] TS 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): tablet a25f2855fb7d4eac9f1c920fdd01e2ba (table rebalancer_dynamic_table_new_name [id=0d23318bd0f5482face4d225e00e7757]) successfully deleted
I20240502 20:46:08.028712 4768 raft_consensus.cc:2932] T 7abfd61f55554042978b50761cbd6ab5 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:08.029306 7017 consensus_queue.cc:1035] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:08.029348 4354 tablet_service.cc:1514] Processing DeleteTablet for tablet 7abfd61f55554042978b50761cbd6ab5 with delete_type TABLET_DATA_TOMBSTONED (TS c92e6e46782849479c8a0c6d82276fb2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38306
I20240502 20:46:08.030383 7297 tablet_replica.cc:330] T 7abfd61f55554042978b50761cbd6ab5 P c92e6e46782849479c8a0c6d82276fb2: stopping tablet replica
I20240502 20:46:08.030602 7297 raft_consensus.cc:2229] T 7abfd61f55554042978b50761cbd6ab5 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:08.030968 7297 raft_consensus.cc:2258] T 7abfd61f55554042978b50761cbd6ab5 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:08.034089 7297 ts_tablet_manager.cc:1875] T 7abfd61f55554042978b50761cbd6ab5 P c92e6e46782849479c8a0c6d82276fb2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:08.035930 7297 ts_tablet_manager.cc:1888] T 7abfd61f55554042978b50761cbd6ab5 P c92e6e46782849479c8a0c6d82276fb2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:46:08.036075 7297 log.cc:1196] T 7abfd61f55554042978b50761cbd6ab5 P c92e6e46782849479c8a0c6d82276fb2: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal/wals/7abfd61f55554042978b50761cbd6ab5
I20240502 20:46:08.036602 8168 raft_consensus.cc:2932] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:08.036823 4222 catalog_manager.cc:4830] TS c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827): tablet 7abfd61f55554042978b50761cbd6ab5 (table rebalance_test_table_13 [id=c22b8adbea7b4dc49a4794a5d8f81f9b]) successfully deleted
I20240502 20:46:08.037895 6702 raft_consensus.cc:2932] T 00eb91e624554baf918e96b098019be8 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:08.038431 4506 raft_consensus.cc:2932] T 00eb91e624554baf918e96b098019be8 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:08.038522 4769 raft_consensus.cc:2932] T 00eb91e624554baf918e96b098019be8 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:08.041339 4234 catalog_manager.cc:5484] T 00eb91e624554baf918e96b098019be8 P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: config changed from index 4 to 5, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:08.044446 8168 raft_consensus.cc:1050] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91: attempting to promote NON_VOTER a47e5c56f13843018ac584914a8d4707 to VOTER
I20240502 20:46:08.044821 8168 consensus_queue.cc:237] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } }
I20240502 20:46:08.046116 6957 raft_consensus.cc:1261] T 8565ebbd21dd4f708a0254d1ed2d6670 P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:08.046202 4506 raft_consensus.cc:1261] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:08.046238 4637 consensus_queue.cc:237] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } }
I20240502 20:46:08.046433 4769 raft_consensus.cc:1261] T 8565ebbd21dd4f708a0254d1ed2d6670 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:08.046864 7447 consensus_queue.cc:1035] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:08.047011 8168 consensus_queue.cc:1035] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:08.047161 7017 consensus_queue.cc:1035] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:08.048485 6702 raft_consensus.cc:1261] T 00eb91e624554baf918e96b098019be8 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:08.048628 4768 raft_consensus.cc:1261] T 00eb91e624554baf918e96b098019be8 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:08.048955 8168 consensus_queue.cc:1035] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:08.049199 7017 consensus_queue.cc:1035] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:08.051508 7017 raft_consensus.cc:2932] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:08.053233 4768 raft_consensus.cc:2932] T 8565ebbd21dd4f708a0254d1ed2d6670 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:08.054714 4233 catalog_manager.cc:5484] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index 4 to 5, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:08.056324 7447 raft_consensus.cc:2932] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:08.058183 4219 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 00eb91e624554baf918e96b098019be8 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:08.058187 4768 raft_consensus.cc:2932] T 00eb91e624554baf918e96b098019be8 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:08.058537 6702 raft_consensus.cc:2932] T 00eb91e624554baf918e96b098019be8 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:08.058544 4506 raft_consensus.cc:2932] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:08.061123 4233 catalog_manager.cc:5484] T 00eb91e624554baf918e96b098019be8 P ed47c023452a497192f35d2f617bf9bd reported cstate change: config changed from index 5 to 6, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:08.064344 4487 tablet_service.cc:1514] Processing DeleteTablet for tablet 00eb91e624554baf918e96b098019be8 with delete_type TABLET_DATA_TOMBSTONED (TS c3d26eafd51547acb780d4aac5ebe7cc not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48890
I20240502 20:46:08.064561 8141 tablet_replica.cc:330] T 00eb91e624554baf918e96b098019be8 P c3d26eafd51547acb780d4aac5ebe7cc: stopping tablet replica
I20240502 20:46:08.064615 7937 raft_consensus.cc:1050] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2: attempting to promote NON_VOTER 201d2b6485084e7dad2cbb722af94779 to VOTER
I20240502 20:46:08.064695 8141 raft_consensus.cc:2229] T 00eb91e624554baf918e96b098019be8 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:08.064851 8141 raft_consensus.cc:2258] T 00eb91e624554baf918e96b098019be8 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:08.065197 7937 consensus_queue.cc:237] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } }
I20240502 20:46:08.065905 8141 ts_tablet_manager.cc:1875] T 00eb91e624554baf918e96b098019be8 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:08.066831 4506 raft_consensus.cc:1261] T 8e7e145762bf458a8374bfc503025a17 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:08.067088 7099 raft_consensus.cc:1261] T 8e7e145762bf458a8374bfc503025a17 P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:08.067418 7432 consensus_queue.cc:1035] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:08.067878 7432 consensus_queue.cc:1035] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:08.068210 8141 ts_tablet_manager.cc:1888] T 00eb91e624554baf918e96b098019be8 P c3d26eafd51547acb780d4aac5ebe7cc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:46:08.068285 8141 log.cc:1196] T 00eb91e624554baf918e96b098019be8 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal/wals/00eb91e624554baf918e96b098019be8
I20240502 20:46:08.068867 4221 catalog_manager.cc:4830] TS c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097): tablet 00eb91e624554baf918e96b098019be8 (table rebalance_test_table_3 [id=54d5208bcf1c43998e3ac81026ea9fbe]) successfully deleted
I20240502 20:46:08.069188 4637 raft_consensus.cc:1261] T 8e7e145762bf458a8374bfc503025a17 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:08.069622 7432 consensus_queue.cc:1035] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:08.071513 7432 raft_consensus.cc:2932] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:08.073500 4506 raft_consensus.cc:2932] T 8e7e145762bf458a8374bfc503025a17 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:08.073992 4637 raft_consensus.cc:2932] T 8e7e145762bf458a8374bfc503025a17 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:08.074373 4234 catalog_manager.cc:5484] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2 reported cstate change: config changed from index 4 to 5, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c92e6e46782849479c8a0c6d82276fb2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:08.075850 7099 raft_consensus.cc:2932] T 8e7e145762bf458a8374bfc503025a17 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:08.078401 6957 raft_consensus.cc:2932] T 8565ebbd21dd4f708a0254d1ed2d6670 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:08.081161 4637 consensus_queue.cc:237] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } }
I20240502 20:46:08.081745 8026 consensus_queue.cc:1035] T 6ed088a162a14d37b5da21ebb468a03b P 201d2b6485084e7dad2cbb722af94779 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:08.082580 4508 raft_consensus.cc:1261] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:08.082631 6957 raft_consensus.cc:1261] T 8565ebbd21dd4f708a0254d1ed2d6670 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:08.083053 8168 consensus_queue.cc:1035] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:08.083367 8168 consensus_queue.cc:1035] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:08.084957 7017 raft_consensus.cc:2932] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:08.085386 4507 raft_consensus.cc:2932] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:08.087306 4219 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8565ebbd21dd4f708a0254d1ed2d6670 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:08.087260 4377 consensus_queue.cc:237] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } }
I20240502 20:46:08.087231 6957 raft_consensus.cc:2932] T 8565ebbd21dd4f708a0254d1ed2d6670 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:08.089650 7099 raft_consensus.cc:1261] T 8e7e145762bf458a8374bfc503025a17 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:08.089831 4228 catalog_manager.cc:5484] T 8565ebbd21dd4f708a0254d1ed2d6670 P a47e5c56f13843018ac584914a8d4707 reported cstate change: config changed from index 5 to 6, VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) evicted. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:08.091271 6138 consensus_queue.cc:1035] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:08.091425 4507 raft_consensus.cc:1261] T 8e7e145762bf458a8374bfc503025a17 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:08.092101 8181 consensus_queue.cc:1035] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:08.095530 7946 consensus_queue.cc:1035] T 6ed088a162a14d37b5da21ebb468a03b P 201d2b6485084e7dad2cbb722af94779 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:08.095213 8185 raft_consensus.cc:2932] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:08.096127 4507 raft_consensus.cc:2932] T 8e7e145762bf458a8374bfc503025a17 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:08.099081 4222 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8e7e145762bf458a8374bfc503025a17 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:08.099560 7099 raft_consensus.cc:2932] T 8e7e145762bf458a8374bfc503025a17 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:08.100672 4750 tablet_service.cc:1514] Processing DeleteTablet for tablet 8565ebbd21dd4f708a0254d1ed2d6670 with delete_type TABLET_DATA_TOMBSTONED (TS ed47c023452a497192f35d2f617bf9bd not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38526
I20240502 20:46:08.100939 7299 tablet_replica.cc:330] T 8565ebbd21dd4f708a0254d1ed2d6670 P ed47c023452a497192f35d2f617bf9bd: stopping tablet replica
I20240502 20:46:08.101130 7299 raft_consensus.cc:2229] T 8565ebbd21dd4f708a0254d1ed2d6670 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:08.101338 7299 raft_consensus.cc:2258] T 8565ebbd21dd4f708a0254d1ed2d6670 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:08.102123 7299 ts_tablet_manager.cc:1875] T 8565ebbd21dd4f708a0254d1ed2d6670 P ed47c023452a497192f35d2f617bf9bd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:08.102032 4233 catalog_manager.cc:5484] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2 reported cstate change: config changed from index 5 to 6, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New cstate: current_term: 1 leader_uuid: "c92e6e46782849479c8a0c6d82276fb2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:08.104434 7299 ts_tablet_manager.cc:1888] T 8565ebbd21dd4f708a0254d1ed2d6670 P ed47c023452a497192f35d2f617bf9bd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:46:08.104512 7299 log.cc:1196] T 8565ebbd21dd4f708a0254d1ed2d6670 P ed47c023452a497192f35d2f617bf9bd: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal/wals/8565ebbd21dd4f708a0254d1ed2d6670
I20240502 20:46:08.105144 4222 catalog_manager.cc:4830] TS ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193): tablet 8565ebbd21dd4f708a0254d1ed2d6670 (table rebalance_test_table_11 [id=105c28a926194705956796da40cf3dab]) successfully deleted
I20240502 20:46:08.107053 4618 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e7e145762bf458a8374bfc503025a17 with delete_type TABLET_DATA_TOMBSTONED (TS 923cfab926e7443f8ffd7b21ad0a9d91 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57068
I20240502 20:46:08.107247 7913 tablet_replica.cc:330] T 8e7e145762bf458a8374bfc503025a17 P 923cfab926e7443f8ffd7b21ad0a9d91: stopping tablet replica
I20240502 20:46:08.107385 7913 raft_consensus.cc:2229] T 8e7e145762bf458a8374bfc503025a17 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:08.107529 7913 raft_consensus.cc:2258] T 8e7e145762bf458a8374bfc503025a17 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:08.108470 7913 ts_tablet_manager.cc:1875] T 8e7e145762bf458a8374bfc503025a17 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:08.110334 7913 ts_tablet_manager.cc:1888] T 8e7e145762bf458a8374bfc503025a17 P 923cfab926e7443f8ffd7b21ad0a9d91: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:46:08.110394 7913 log.cc:1196] T 8e7e145762bf458a8374bfc503025a17 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal/wals/8e7e145762bf458a8374bfc503025a17
I20240502 20:46:08.111066 4219 catalog_manager.cc:4830] TS 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141): tablet 8e7e145762bf458a8374bfc503025a17 (table rebalance_test_table_15 [id=57f2e4fb3cca4e408bd33dac3f4b9d0c]) successfully deleted
I20240502 20:46:08.117969 4228 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41812:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 20:46:08.120148 4507 consensus_queue.cc:237] T c0c95d50b8f64d829b7edbcd6f7c931e P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } }
I20240502 20:46:08.121927 4768 raft_consensus.cc:1261] T c0c95d50b8f64d829b7edbcd6f7c931e P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 20:46:08.122033 4442 consensus_peers.cc:485] T c0c95d50b8f64d829b7edbcd6f7c931e P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867): Couldn't send request to peer a4bf1df7fa6b4948be502c182bfbb833. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0c95d50b8f64d829b7edbcd6f7c931e. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:08.122792 4374 raft_consensus.cc:1261] T c0c95d50b8f64d829b7edbcd6f7c931e P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:08.123034 6375 consensus_queue.cc:1035] T c0c95d50b8f64d829b7edbcd6f7c931e P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:08.123556 4750 tablet_service.cc:1467] Processing CreateTablet for tablet 0dc7d9f64ba2438fb37fbe0b33f4a8be (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4c3872020dc144bd8931994dced22724]), partition=
I20240502 20:46:08.123589 6947 tablet_service.cc:1467] Processing CreateTablet for tablet 0dc7d9f64ba2438fb37fbe0b33f4a8be (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4c3872020dc144bd8931994dced22724]), partition=
I20240502 20:46:08.123811 4750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dc7d9f64ba2438fb37fbe0b33f4a8be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:08.123852 6947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dc7d9f64ba2438fb37fbe0b33f4a8be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:08.123983 6809 tablet_service.cc:1467] Processing CreateTablet for tablet 0dc7d9f64ba2438fb37fbe0b33f4a8be (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4c3872020dc144bd8931994dced22724]), partition=
I20240502 20:46:08.124199 6809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dc7d9f64ba2438fb37fbe0b33f4a8be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:08.124776 6375 consensus_queue.cc:1035] T c0c95d50b8f64d829b7edbcd6f7c931e P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:08.126175 7939 tablet_bootstrap.cc:492] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P a47e5c56f13843018ac584914a8d4707: Bootstrap starting.
I20240502 20:46:08.126333 8014 tablet_bootstrap.cc:492] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap starting.
I20240502 20:46:08.127113 8014 tablet_bootstrap.cc:654] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P a4bf1df7fa6b4948be502c182bfbb833: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:08.127409 7939 tablet_bootstrap.cc:654] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P a47e5c56f13843018ac584914a8d4707: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:08.127317 6375 raft_consensus.cc:2932] T c0c95d50b8f64d829b7edbcd6f7c931e P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } } }
I20240502 20:46:08.128573 8198 tablet_bootstrap.cc:492] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:46:08.128813 8014 tablet_bootstrap.cc:492] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P a4bf1df7fa6b4948be502c182bfbb833: No bootstrap required, opened a new log
I20240502 20:46:08.128912 8014 ts_tablet_manager.cc:1367] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P a4bf1df7fa6b4948be502c182bfbb833: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240502 20:46:08.129432 8014 raft_consensus.cc:348] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:08.129550 8014 raft_consensus.cc:374] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:08.129582 8014 raft_consensus.cc:729] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Initialized, Role: FOLLOWER
I20240502 20:46:08.129603 8198 tablet_bootstrap.cc:654] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:08.129685 8014 consensus_queue.cc:260] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P a4bf1df7fa6b4948be502c182bfbb833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:08.129962 8014 ts_tablet_manager.cc:1398] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P a4bf1df7fa6b4948be502c182bfbb833: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:08.131716 4374 raft_consensus.cc:2932] T c0c95d50b8f64d829b7edbcd6f7c931e P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } } }
I20240502 20:46:08.132051 8198 tablet_bootstrap.cc:492] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:46:08.132186 8198 ts_tablet_manager.cc:1367] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 20:46:08.132530 4768 raft_consensus.cc:2932] T c0c95d50b8f64d829b7edbcd6f7c931e P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } } }
I20240502 20:46:08.132835 8198 raft_consensus.cc:348] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:08.133051 8198 raft_consensus.cc:374] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:08.133132 8198 raft_consensus.cc:729] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:46:08.133296 8198 consensus_queue.cc:260] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:08.133767 8198 ts_tablet_manager.cc:1398] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:08.135097 4233 catalog_manager.cc:5484] T c0c95d50b8f64d829b7edbcd6f7c931e P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: config changed from index -1 to 4, NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) added. New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:08.138533 7939 tablet_bootstrap.cc:492] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P a47e5c56f13843018ac584914a8d4707: No bootstrap required, opened a new log
I20240502 20:46:08.138649 7939 ts_tablet_manager.cc:1367] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P a47e5c56f13843018ac584914a8d4707: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.001s
I20240502 20:46:08.139210 7939 raft_consensus.cc:348] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:08.139343 7939 raft_consensus.cc:374] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:08.139389 7939 raft_consensus.cc:729] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Initialized, Role: FOLLOWER
I20240502 20:46:08.139528 7939 consensus_queue.cc:260] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P a47e5c56f13843018ac584914a8d4707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:08.139834 7939 ts_tablet_manager.cc:1398] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P a47e5c56f13843018ac584914a8d4707: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:08.144409 4507 consensus_queue.cc:237] T 66b8f583dfd24c2aa5a59457a8e58fbb P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } }
I20240502 20:46:08.145977 4768 raft_consensus.cc:1261] T 66b8f583dfd24c2aa5a59457a8e58fbb P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:08.146009 4374 raft_consensus.cc:1261] T 66b8f583dfd24c2aa5a59457a8e58fbb P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:08.146505 7404 consensus_queue.cc:1035] T 66b8f583dfd24c2aa5a59457a8e58fbb P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 20:46:08.146986 4444 consensus_peers.cc:485] T 66b8f583dfd24c2aa5a59457a8e58fbb P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 66b8f583dfd24c2aa5a59457a8e58fbb. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:08.147179 7404 consensus_queue.cc:1035] T 66b8f583dfd24c2aa5a59457a8e58fbb P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:08.148665 8146 raft_consensus.cc:2932] T 66b8f583dfd24c2aa5a59457a8e58fbb P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } } }
I20240502 20:46:08.149194 4374 raft_consensus.cc:2932] T 66b8f583dfd24c2aa5a59457a8e58fbb P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } } }
I20240502 20:46:08.149605 4769 raft_consensus.cc:2932] T 66b8f583dfd24c2aa5a59457a8e58fbb P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } } }
I20240502 20:46:08.152504 4228 catalog_manager.cc:5484] T 66b8f583dfd24c2aa5a59457a8e58fbb P ed47c023452a497192f35d2f617bf9bd reported cstate change: config changed from index -1 to 4, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } } }
I20240502 20:46:08.183678 6613 raft_consensus.cc:1050] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91: attempting to promote NON_VOTER a47e5c56f13843018ac584914a8d4707 to VOTER
I20240502 20:46:08.184093 6613 consensus_queue.cc:237] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } }
I20240502 20:46:08.185812 4768 raft_consensus.cc:1261] T 428ffbac75db466191cb3bef63b4eb79 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:08.185885 4374 raft_consensus.cc:1261] T 428ffbac75db466191cb3bef63b4eb79 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:08.186229 6957 raft_consensus.cc:1261] T 428ffbac75db466191cb3bef63b4eb79 P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:08.186254 8168 consensus_queue.cc:1035] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:08.186650 8168 consensus_queue.cc:1035] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:08.186981 8168 consensus_queue.cc:1035] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:08.189227 8168 raft_consensus.cc:2932] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:08.189735 4769 raft_consensus.cc:2932] T 428ffbac75db466191cb3bef63b4eb79 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:08.191192 4374 raft_consensus.cc:2932] T 428ffbac75db466191cb3bef63b4eb79 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:08.192013 4233 catalog_manager.cc:5484] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index 4 to 5, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:08.194188 6957 raft_consensus.cc:2932] T 428ffbac75db466191cb3bef63b4eb79 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:08.199512 4637 consensus_queue.cc:237] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } }
I20240502 20:46:08.200757 6957 raft_consensus.cc:1261] T 428ffbac75db466191cb3bef63b4eb79 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:08.200898 4374 raft_consensus.cc:1261] T 428ffbac75db466191cb3bef63b4eb79 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:08.201195 8236 consensus_queue.cc:1035] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:08.201381 8168 consensus_queue.cc:1035] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:08.203166 8236 raft_consensus.cc:2932] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:08.203492 4374 raft_consensus.cc:2932] T 428ffbac75db466191cb3bef63b4eb79 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:08.205612 4219 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 428ffbac75db466191cb3bef63b4eb79 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:08.210417 4233 catalog_manager.cc:5484] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index 5 to 6, VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) evicted. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:08.211253 6957 raft_consensus.cc:2932] T 428ffbac75db466191cb3bef63b4eb79 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:08.213126 4750 tablet_service.cc:1514] Processing DeleteTablet for tablet 428ffbac75db466191cb3bef63b4eb79 with delete_type TABLET_DATA_TOMBSTONED (TS ed47c023452a497192f35d2f617bf9bd not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38526
I20240502 20:46:08.213408 7299 tablet_replica.cc:330] T 428ffbac75db466191cb3bef63b4eb79 P ed47c023452a497192f35d2f617bf9bd: stopping tablet replica
I20240502 20:46:08.213584 7299 raft_consensus.cc:2229] T 428ffbac75db466191cb3bef63b4eb79 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:08.213753 7299 raft_consensus.cc:2258] T 428ffbac75db466191cb3bef63b4eb79 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:08.214742 7299 ts_tablet_manager.cc:1875] T 428ffbac75db466191cb3bef63b4eb79 P ed47c023452a497192f35d2f617bf9bd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:08.216969 7299 ts_tablet_manager.cc:1888] T 428ffbac75db466191cb3bef63b4eb79 P ed47c023452a497192f35d2f617bf9bd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:46:08.217056 7299 log.cc:1196] T 428ffbac75db466191cb3bef63b4eb79 P ed47c023452a497192f35d2f617bf9bd: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal/wals/428ffbac75db466191cb3bef63b4eb79
I20240502 20:46:08.217666 4222 catalog_manager.cc:4830] TS ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193): tablet 428ffbac75db466191cb3bef63b4eb79 (table rebalance_test_table_7 [id=1e32a0fff569475cbd44b12c4fd5e8e6]) successfully deleted
I20240502 20:46:08.236725 6241 raft_consensus.cc:1050] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd: attempting to promote NON_VOTER 7845c397230d49358ba2fad9db53f97c to VOTER
I20240502 20:46:08.237193 6241 consensus_queue.cc:237] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } }
I20240502 20:46:08.238562 4637 raft_consensus.cc:1261] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:08.239053 7996 consensus_queue.cc:1035] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:08.239308 6702 raft_consensus.cc:1261] T d37fab152a8a44868d3160ec3057bb98 P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:08.240571 4507 raft_consensus.cc:1261] T d37fab152a8a44868d3160ec3057bb98 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:08.241089 7404 raft_consensus.cc:1050] T 61f8c8d2925d46bdad8e903b48c3046e P c3d26eafd51547acb780d4aac5ebe7cc: attempting to promote NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 to VOTER
I20240502 20:46:08.241461 7404 consensus_queue.cc:237] T 61f8c8d2925d46bdad8e903b48c3046e P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } }
I20240502 20:46:08.243224 7318 consensus_queue.cc:1035] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:08.243511 7318 consensus_queue.cc:1035] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:08.245024 4374 raft_consensus.cc:1261] T 61f8c8d2925d46bdad8e903b48c3046e P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:08.245149 4638 raft_consensus.cc:1261] T 61f8c8d2925d46bdad8e903b48c3046e P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:08.245438 7921 consensus_queue.cc:1035] T 61f8c8d2925d46bdad8e903b48c3046e P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:08.245599 7967 consensus_queue.cc:1035] T 61f8c8d2925d46bdad8e903b48c3046e P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:08.246644 6831 raft_consensus.cc:1261] T 61f8c8d2925d46bdad8e903b48c3046e P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEARNER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:08.247112 7921 consensus_queue.cc:1035] T 61f8c8d2925d46bdad8e903b48c3046e P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:08.251127 7967 raft_consensus.cc:2932] T 61f8c8d2925d46bdad8e903b48c3046e P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:08.251610 4638 raft_consensus.cc:2932] T 61f8c8d2925d46bdad8e903b48c3046e P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:08.251838 8111 raft_consensus.cc:2932] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:08.252252 4637 raft_consensus.cc:2932] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:08.254748 4233 catalog_manager.cc:5484] T 61f8c8d2925d46bdad8e903b48c3046e P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index 4 to 5, a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:08.254825 6702 raft_consensus.cc:2932] T d37fab152a8a44868d3160ec3057bb98 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:08.254941 4229 catalog_manager.cc:5484] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd reported cstate change: config changed from index 4 to 5, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:08.256922 4507 raft_consensus.cc:2932] T d37fab152a8a44868d3160ec3057bb98 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:08.257994 6831 raft_consensus.cc:2932] T 61f8c8d2925d46bdad8e903b48c3046e P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:08.261009 4769 consensus_queue.cc:237] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } }
I20240502 20:46:08.262084 6702 raft_consensus.cc:1261] T d37fab152a8a44868d3160ec3057bb98 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:08.262243 4637 raft_consensus.cc:1261] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:08.262466 6241 consensus_queue.cc:1035] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:08.262715 7318 consensus_queue.cc:1035] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:08.263753 8131 consensus_queue.cc:1035] T 3d938d01d5154590a9262acd54094ad2 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:08.264719 7318 raft_consensus.cc:2932] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:08.267526 4637 raft_consensus.cc:2932] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:08.267863 4222 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d37fab152a8a44868d3160ec3057bb98 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:08.269270 4374 raft_consensus.cc:2932] T 61f8c8d2925d46bdad8e903b48c3046e P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:08.269445 4507 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5a36bf6e8e6e43b2b47b6d6357b0d445"
dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46500
I20240502 20:46:08.269603 4507 raft_consensus.cc:595] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Received request to transfer leadership
I20240502 20:46:08.271200 4234 catalog_manager.cc:5484] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd reported cstate change: config changed from index 5 to 6, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:08.272131 4507 consensus_queue.cc:237] T 61f8c8d2925d46bdad8e903b48c3046e P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } }
I20240502 20:46:08.273949 6831 raft_consensus.cc:1261] T 61f8c8d2925d46bdad8e903b48c3046e P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:08.274874 7165 consensus_queue.cc:1035] T 3d938d01d5154590a9262acd54094ad2 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:08.275470 8146 consensus_queue.cc:1035] T 61f8c8d2925d46bdad8e903b48c3046e P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:08.277247 4374 raft_consensus.cc:1261] T 61f8c8d2925d46bdad8e903b48c3046e P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:08.277480 6702 raft_consensus.cc:2932] T d37fab152a8a44868d3160ec3057bb98 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:08.277688 7921 consensus_queue.cc:1035] T 61f8c8d2925d46bdad8e903b48c3046e P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:08.277880 8146 raft_consensus.cc:1050] T f09dcfea3eb347e687f3c72f4ccc6929 P c3d26eafd51547acb780d4aac5ebe7cc: attempting to promote NON_VOTER a47e5c56f13843018ac584914a8d4707 to VOTER
I20240502 20:46:08.278252 8146 consensus_queue.cc:237] T f09dcfea3eb347e687f3c72f4ccc6929 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } }
I20240502 20:46:08.278740 4487 tablet_service.cc:1514] Processing DeleteTablet for tablet d37fab152a8a44868d3160ec3057bb98 with delete_type TABLET_DATA_TOMBSTONED (TS c3d26eafd51547acb780d4aac5ebe7cc not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48890
I20240502 20:46:08.278934 8141 tablet_replica.cc:330] T d37fab152a8a44868d3160ec3057bb98 P c3d26eafd51547acb780d4aac5ebe7cc: stopping tablet replica
I20240502 20:46:08.279073 8141 raft_consensus.cc:2229] T d37fab152a8a44868d3160ec3057bb98 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:08.279227 8141 raft_consensus.cc:2258] T d37fab152a8a44868d3160ec3057bb98 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:08.280066 7921 raft_consensus.cc:2932] T 61f8c8d2925d46bdad8e903b48c3046e P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:08.280192 8141 ts_tablet_manager.cc:1875] T d37fab152a8a44868d3160ec3057bb98 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:08.280419 6831 raft_consensus.cc:2932] T 61f8c8d2925d46bdad8e903b48c3046e P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:08.282258 8141 ts_tablet_manager.cc:1888] T d37fab152a8a44868d3160ec3057bb98 P c3d26eafd51547acb780d4aac5ebe7cc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:46:08.282325 8141 log.cc:1196] T d37fab152a8a44868d3160ec3057bb98 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal/wals/d37fab152a8a44868d3160ec3057bb98
I20240502 20:46:08.285141 4769 raft_consensus.cc:1261] T f09dcfea3eb347e687f3c72f4ccc6929 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:08.285141 4374 raft_consensus.cc:1261] T f09dcfea3eb347e687f3c72f4ccc6929 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:08.285035 4233 catalog_manager.cc:5484] T 61f8c8d2925d46bdad8e903b48c3046e P a4bf1df7fa6b4948be502c182bfbb833 reported cstate change: config changed from index 5 to 6, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:08.285928 6957 raft_consensus.cc:1261] T f09dcfea3eb347e687f3c72f4ccc6929 P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:08.286051 4221 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 61f8c8d2925d46bdad8e903b48c3046e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:08.286197 4221 catalog_manager.cc:4830] TS c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097): tablet d37fab152a8a44868d3160ec3057bb98 (table rebalance_test_table_5 [id=5c60c96389ff4cf3bbd5edb6a694c70d]) successfully deleted
I20240502 20:46:08.287989 7404 consensus_queue.cc:1035] T f09dcfea3eb347e687f3c72f4ccc6929 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:08.288028 4374 raft_consensus.cc:2932] T 61f8c8d2925d46bdad8e903b48c3046e P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:08.288300 6375 consensus_queue.cc:1035] T f09dcfea3eb347e687f3c72f4ccc6929 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:08.289549 7921 consensus_queue.cc:1035] T f09dcfea3eb347e687f3c72f4ccc6929 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:08.294458 8146 raft_consensus.cc:2932] T f09dcfea3eb347e687f3c72f4ccc6929 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:08.294911 6957 raft_consensus.cc:2932] T f09dcfea3eb347e687f3c72f4ccc6929 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:08.295920 4618 tablet_service.cc:1514] Processing DeleteTablet for tablet 61f8c8d2925d46bdad8e903b48c3046e with delete_type TABLET_DATA_TOMBSTONED (TS 923cfab926e7443f8ffd7b21ad0a9d91 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57068
I20240502 20:46:08.296139 7913 tablet_replica.cc:330] T 61f8c8d2925d46bdad8e903b48c3046e P 923cfab926e7443f8ffd7b21ad0a9d91: stopping tablet replica
I20240502 20:46:08.296314 7913 raft_consensus.cc:2229] T 61f8c8d2925d46bdad8e903b48c3046e P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:08.296489 7913 raft_consensus.cc:2258] T 61f8c8d2925d46bdad8e903b48c3046e P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:08.297222 7913 ts_tablet_manager.cc:1875] T 61f8c8d2925d46bdad8e903b48c3046e P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:08.297255 4769 raft_consensus.cc:2932] T f09dcfea3eb347e687f3c72f4ccc6929 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:08.297520 4228 catalog_manager.cc:5484] T f09dcfea3eb347e687f3c72f4ccc6929 P a47e5c56f13843018ac584914a8d4707 reported cstate change: config changed from index 4 to 5, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:08.299412 7913 ts_tablet_manager.cc:1888] T 61f8c8d2925d46bdad8e903b48c3046e P 923cfab926e7443f8ffd7b21ad0a9d91: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:46:08.299466 7913 log.cc:1196] T 61f8c8d2925d46bdad8e903b48c3046e P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal/wals/61f8c8d2925d46bdad8e903b48c3046e
I20240502 20:46:08.300850 4219 catalog_manager.cc:4830] TS 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141): tablet 61f8c8d2925d46bdad8e903b48c3046e (table rebalance_test_table_11 [id=105c28a926194705956796da40cf3dab]) successfully deleted
I20240502 20:46:08.301024 4377 raft_consensus.cc:2932] T f09dcfea3eb347e687f3c72f4ccc6929 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:08.303664 4507 consensus_queue.cc:237] T f09dcfea3eb347e687f3c72f4ccc6929 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } }
I20240502 20:46:08.304781 6957 raft_consensus.cc:1261] T f09dcfea3eb347e687f3c72f4ccc6929 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:08.304800 4377 raft_consensus.cc:1261] T f09dcfea3eb347e687f3c72f4ccc6929 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:08.305218 8146 consensus_queue.cc:1035] T f09dcfea3eb347e687f3c72f4ccc6929 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:08.305578 7967 consensus_queue.cc:1035] T f09dcfea3eb347e687f3c72f4ccc6929 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:08.306684 4233 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240502 20:46:08.306797 4233 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240502 20:46:08.307183 6375 raft_consensus.cc:2932] T f09dcfea3eb347e687f3c72f4ccc6929 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:08.307569 4377 raft_consensus.cc:2932] T f09dcfea3eb347e687f3c72f4ccc6929 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:08.309933 4221 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f09dcfea3eb347e687f3c72f4ccc6929 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:08.310209 7996 raft_consensus.cc:482] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:08.310318 7996 raft_consensus.cc:504] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:08.310739 4233 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: Sending DeleteTablet for 4 replicas of tablet c0c95d50b8f64d829b7edbcd6f7c931e
I20240502 20:46:08.310806 7996 leader_election.cc:288] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867)
I20240502 20:46:08.311359 4487 tablet_service.cc:1514] Processing DeleteTablet for tablet c0c95d50b8f64d829b7edbcd6f7c931e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:08 UTC) from {username='slave'} at 127.0.0.1:48890
I20240502 20:46:08.311429 4354 tablet_service.cc:1514] Processing DeleteTablet for tablet c0c95d50b8f64d829b7edbcd6f7c931e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:08 UTC) from {username='slave'} at 127.0.0.1:38306
I20240502 20:46:08.311532 8141 tablet_replica.cc:330] T c0c95d50b8f64d829b7edbcd6f7c931e P c3d26eafd51547acb780d4aac5ebe7cc: stopping tablet replica
I20240502 20:46:08.311708 8141 raft_consensus.cc:2229] T c0c95d50b8f64d829b7edbcd6f7c931e P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:08.311704 4750 tablet_service.cc:1514] Processing DeleteTablet for tablet c0c95d50b8f64d829b7edbcd6f7c931e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:08 UTC) from {username='slave'} at 127.0.0.1:38526
I20240502 20:46:08.311873 7299 tablet_replica.cc:330] T c0c95d50b8f64d829b7edbcd6f7c931e P ed47c023452a497192f35d2f617bf9bd: stopping tablet replica
I20240502 20:46:08.311946 8141 raft_consensus.cc:2258] T c0c95d50b8f64d829b7edbcd6f7c931e P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:08.311923 6831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dc7d9f64ba2438fb37fbe0b33f4a8be" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833" is_pre_election: true
I20240502 20:46:08.312016 7299 raft_consensus.cc:2229] T c0c95d50b8f64d829b7edbcd6f7c931e P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:08.312075 6831 raft_consensus.cc:2454] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 0.
I20240502 20:46:08.312125 7297 tablet_replica.cc:330] T c0c95d50b8f64d829b7edbcd6f7c931e P c92e6e46782849479c8a0c6d82276fb2: stopping tablet replica
I20240502 20:46:08.312151 7299 raft_consensus.cc:2258] T c0c95d50b8f64d829b7edbcd6f7c931e P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:08.312268 7297 raft_consensus.cc:2229] T c0c95d50b8f64d829b7edbcd6f7c931e P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:08.312386 4704 leader_election.cc:302] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4bf1df7fa6b4948be502c182bfbb833, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:46:08.312422 7297 raft_consensus.cc:2258] T c0c95d50b8f64d829b7edbcd6f7c931e P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:08.312552 7996 raft_consensus.cc:2781] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:08.312628 7996 raft_consensus.cc:482] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:08.312667 7996 raft_consensus.cc:3037] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:08.312829 4233 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: Sending DeleteTablet for 3 replicas of tablet 789757df8e7d4284abcd7fcdf7538cf5
I20240502 20:46:08.313117 6957 raft_consensus.cc:2932] T f09dcfea3eb347e687f3c72f4ccc6929 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:08.313299 4354 tablet_service.cc:1514] Processing DeleteTablet for tablet 789757df8e7d4284abcd7fcdf7538cf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:08 UTC) from {username='slave'} at 127.0.0.1:38306
I20240502 20:46:08.313463 7299 ts_tablet_manager.cc:1875] T c0c95d50b8f64d829b7edbcd6f7c931e P ed47c023452a497192f35d2f617bf9bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:08.313710 7996 raft_consensus.cc:504] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:08.314038 7996 leader_election.cc:288] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 election: Requested vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867)
I20240502 20:46:08.314122 4233 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: Sending DeleteTablet for 3 replicas of tablet 9e34e8e17a7e413b97eb614a7df727d5
I20240502 20:46:08.314236 6809 tablet_service.cc:1514] Processing DeleteTablet for tablet c0c95d50b8f64d829b7edbcd6f7c931e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:08 UTC) from {username='slave'} at 127.0.0.1:35290
I20240502 20:46:08.314437 4618 tablet_service.cc:1514] Processing DeleteTablet for tablet 789757df8e7d4284abcd7fcdf7538cf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:08 UTC) from {username='slave'} at 127.0.0.1:57068
I20240502 20:46:08.314534 4354 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e34e8e17a7e413b97eb614a7df727d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:08 UTC) from {username='slave'} at 127.0.0.1:38306
I20240502 20:46:08.314908 7913 tablet_replica.cc:330] T 789757df8e7d4284abcd7fcdf7538cf5 P 923cfab926e7443f8ffd7b21ad0a9d91: stopping tablet replica
I20240502 20:46:08.315035 7913 raft_consensus.cc:2229] T 789757df8e7d4284abcd7fcdf7538cf5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus shutting down.
W20240502 20:46:08.315088 4220 catalog_manager.cc:4797] TS a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867): delete failed for tablet c0c95d50b8f64d829b7edbcd6f7c931e because the tablet was not found. No further retry: Not found: Tablet not found: c0c95d50b8f64d829b7edbcd6f7c931e
I20240502 20:46:08.315188 7913 raft_consensus.cc:2258] T 789757df8e7d4284abcd7fcdf7538cf5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:08.315352 4234 catalog_manager.cc:5484] T f09dcfea3eb347e687f3c72f4ccc6929 P c92e6e46782849479c8a0c6d82276fb2 reported cstate change: config changed from index 5 to 6, VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) evicted. New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:08.315670 7299 ts_tablet_manager.cc:1888] T c0c95d50b8f64d829b7edbcd6f7c931e P ed47c023452a497192f35d2f617bf9bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:08.315742 7299 log.cc:1196] T c0c95d50b8f64d829b7edbcd6f7c931e P ed47c023452a497192f35d2f617bf9bd: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal/wals/c0c95d50b8f64d829b7edbcd6f7c931e
I20240502 20:46:08.315755 7913 ts_tablet_manager.cc:1875] T 789757df8e7d4284abcd7fcdf7538cf5 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:08.316008 7299 ts_tablet_manager.cc:1909] T c0c95d50b8f64d829b7edbcd6f7c931e P ed47c023452a497192f35d2f617bf9bd: Deleting consensus metadata
I20240502 20:46:08.317047 7297 ts_tablet_manager.cc:1875] T c0c95d50b8f64d829b7edbcd6f7c931e P c92e6e46782849479c8a0c6d82276fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:08.317958 7913 ts_tablet_manager.cc:1888] T 789757df8e7d4284abcd7fcdf7538cf5 P 923cfab926e7443f8ffd7b21ad0a9d91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 20:46:08.318024 7913 log.cc:1196] T 789757df8e7d4284abcd7fcdf7538cf5 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal/wals/789757df8e7d4284abcd7fcdf7538cf5
I20240502 20:46:08.318312 7913 ts_tablet_manager.cc:1909] T 789757df8e7d4284abcd7fcdf7538cf5 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting consensus metadata
I20240502 20:46:08.319005 4219 catalog_manager.cc:4830] TS 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141): tablet 789757df8e7d4284abcd7fcdf7538cf5 (table rebalance_test_table_10 [id=e54c8dd95147474b9f91de67bc9c075c]) successfully deleted
I20240502 20:46:08.319208 7297 ts_tablet_manager.cc:1888] T c0c95d50b8f64d829b7edbcd6f7c931e P c92e6e46782849479c8a0c6d82276fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:08.319273 7297 log.cc:1196] T c0c95d50b8f64d829b7edbcd6f7c931e P c92e6e46782849479c8a0c6d82276fb2: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal/wals/c0c95d50b8f64d829b7edbcd6f7c931e
I20240502 20:46:08.319380 7079 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e34e8e17a7e413b97eb614a7df727d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:08 UTC) from {username='slave'} at 127.0.0.1:59162
I20240502 20:46:08.319380 4750 tablet_service.cc:1514] Processing DeleteTablet for tablet 789757df8e7d4284abcd7fcdf7538cf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:08 UTC) from {username='slave'} at 127.0.0.1:38526
I20240502 20:46:08.319537 7297 ts_tablet_manager.cc:1909] T c0c95d50b8f64d829b7edbcd6f7c931e P c92e6e46782849479c8a0c6d82276fb2: Deleting consensus metadata
I20240502 20:46:08.319576 8137 tablet_replica.cc:330] T 9e34e8e17a7e413b97eb614a7df727d5 P 201d2b6485084e7dad2cbb722af94779: stopping tablet replica
I20240502 20:46:08.319696 8137 raft_consensus.cc:2229] T 9e34e8e17a7e413b97eb614a7df727d5 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:08.319831 8137 raft_consensus.cc:2258] T 9e34e8e17a7e413b97eb614a7df727d5 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:08.319950 4750 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e34e8e17a7e413b97eb614a7df727d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:08 UTC) from {username='slave'} at 127.0.0.1:38526
I20240502 20:46:08.319996 4749 tablet_service.cc:1514] Processing DeleteTablet for tablet f09dcfea3eb347e687f3c72f4ccc6929 with delete_type TABLET_DATA_TOMBSTONED (TS ed47c023452a497192f35d2f617bf9bd not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38526
I20240502 20:46:08.320120 7299 tablet_replica.cc:330] T 789757df8e7d4284abcd7fcdf7538cf5 P ed47c023452a497192f35d2f617bf9bd: stopping tablet replica
I20240502 20:46:08.320163 7297 tablet_replica.cc:330] T 789757df8e7d4284abcd7fcdf7538cf5 P c92e6e46782849479c8a0c6d82276fb2: stopping tablet replica
I20240502 20:46:08.320290 7297 raft_consensus.cc:2229] T 789757df8e7d4284abcd7fcdf7538cf5 P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:08.320405 6831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dc7d9f64ba2438fb37fbe0b33f4a8be" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833"
I20240502 20:46:08.320467 7297 raft_consensus.cc:2258] T 789757df8e7d4284abcd7fcdf7538cf5 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:08.320537 6831 raft_consensus.cc:3037] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:08.320541 7299 raft_consensus.cc:2229] T 789757df8e7d4284abcd7fcdf7538cf5 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:08.320679 7299 raft_consensus.cc:2258] T 789757df8e7d4284abcd7fcdf7538cf5 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:08.320960 7297 ts_tablet_manager.cc:1875] T 789757df8e7d4284abcd7fcdf7538cf5 P c92e6e46782849479c8a0c6d82276fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:08.321139 7299 ts_tablet_manager.cc:1875] T 789757df8e7d4284abcd7fcdf7538cf5 P ed47c023452a497192f35d2f617bf9bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:08.321599 6831 raft_consensus.cc:2454] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 1.
I20240502 20:46:08.322214 8137 ts_tablet_manager.cc:1875] T 9e34e8e17a7e413b97eb614a7df727d5 P 201d2b6485084e7dad2cbb722af94779: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:08.322227 8141 ts_tablet_manager.cc:1875] T c0c95d50b8f64d829b7edbcd6f7c931e P c3d26eafd51547acb780d4aac5ebe7cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:08.322957 7299 ts_tablet_manager.cc:1888] T 789757df8e7d4284abcd7fcdf7538cf5 P ed47c023452a497192f35d2f617bf9bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 20:46:08.323005 7299 log.cc:1196] T 789757df8e7d4284abcd7fcdf7538cf5 P ed47c023452a497192f35d2f617bf9bd: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal/wals/789757df8e7d4284abcd7fcdf7538cf5
I20240502 20:46:08.323189 7299 ts_tablet_manager.cc:1909] T 789757df8e7d4284abcd7fcdf7538cf5 P ed47c023452a497192f35d2f617bf9bd: Deleting consensus metadata
I20240502 20:46:08.323881 7299 tablet_replica.cc:330] T 9e34e8e17a7e413b97eb614a7df727d5 P ed47c023452a497192f35d2f617bf9bd: stopping tablet replica
I20240502 20:46:08.324012 7299 raft_consensus.cc:2229] T 9e34e8e17a7e413b97eb614a7df727d5 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:08.324190 7299 raft_consensus.cc:2258] T 9e34e8e17a7e413b97eb614a7df727d5 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:08.324380 8137 ts_tablet_manager.cc:1888] T 9e34e8e17a7e413b97eb614a7df727d5 P 201d2b6485084e7dad2cbb722af94779: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240502 20:46:08.324455 8137 log.cc:1196] T 9e34e8e17a7e413b97eb614a7df727d5 P 201d2b6485084e7dad2cbb722af94779: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/wal/wals/9e34e8e17a7e413b97eb614a7df727d5
I20240502 20:46:08.324733 8137 ts_tablet_manager.cc:1909] T 9e34e8e17a7e413b97eb614a7df727d5 P 201d2b6485084e7dad2cbb722af94779: Deleting consensus metadata
I20240502 20:46:08.325273 4704 leader_election.cc:302] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4bf1df7fa6b4948be502c182bfbb833, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:46:08.325294 8141 ts_tablet_manager.cc:1888] T c0c95d50b8f64d829b7edbcd6f7c931e P c3d26eafd51547acb780d4aac5ebe7cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:08.325373 8141 log.cc:1196] T c0c95d50b8f64d829b7edbcd6f7c931e P c3d26eafd51547acb780d4aac5ebe7cc: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal/wals/c0c95d50b8f64d829b7edbcd6f7c931e
I20240502 20:46:08.325644 8141 ts_tablet_manager.cc:1909] T c0c95d50b8f64d829b7edbcd6f7c931e P c3d26eafd51547acb780d4aac5ebe7cc: Deleting consensus metadata
I20240502 20:46:08.325687 7318 raft_consensus.cc:2781] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:08.325793 7318 raft_consensus.cc:686] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Becoming Leader. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Running, Role: LEADER
I20240502 20:46:08.325908 7318 consensus_queue.cc:237] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:08.326160 7310 consensus_queue.cc:1035] T 747e7331d7ad4dd4b132672254037e1b P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:08.326390 4221 catalog_manager.cc:4830] TS c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097): tablet c0c95d50b8f64d829b7edbcd6f7c931e (table rebalance_test_table_10 [id=e54c8dd95147474b9f91de67bc9c075c]) successfully deleted
I20240502 20:46:08.328016 7297 ts_tablet_manager.cc:1888] T 789757df8e7d4284abcd7fcdf7538cf5 P c92e6e46782849479c8a0c6d82276fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 20:46:08.328083 7297 log.cc:1196] T 789757df8e7d4284abcd7fcdf7538cf5 P c92e6e46782849479c8a0c6d82276fb2: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal/wals/789757df8e7d4284abcd7fcdf7538cf5
I20240502 20:46:08.328343 7297 ts_tablet_manager.cc:1909] T 789757df8e7d4284abcd7fcdf7538cf5 P c92e6e46782849479c8a0c6d82276fb2: Deleting consensus metadata
I20240502 20:46:08.328905 7297 tablet_replica.cc:330] T 9e34e8e17a7e413b97eb614a7df727d5 P c92e6e46782849479c8a0c6d82276fb2: stopping tablet replica
I20240502 20:46:08.329063 7297 raft_consensus.cc:2229] T 9e34e8e17a7e413b97eb614a7df727d5 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:08.329227 7297 raft_consensus.cc:2258] T 9e34e8e17a7e413b97eb614a7df727d5 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:08.329747 7299 ts_tablet_manager.cc:1875] T 9e34e8e17a7e413b97eb614a7df727d5 P ed47c023452a497192f35d2f617bf9bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:08.329957 4222 catalog_manager.cc:4830] TS c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827): tablet c0c95d50b8f64d829b7edbcd6f7c931e (table rebalance_test_table_10 [id=e54c8dd95147474b9f91de67bc9c075c]) successfully deleted
I20240502 20:46:08.330145 4222 catalog_manager.cc:4830] TS c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827): tablet 789757df8e7d4284abcd7fcdf7538cf5 (table rebalance_test_table_10 [id=e54c8dd95147474b9f91de67bc9c075c]) successfully deleted
I20240502 20:46:08.330278 4222 catalog_manager.cc:4830] TS ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193): tablet c0c95d50b8f64d829b7edbcd6f7c931e (table rebalance_test_table_10 [id=e54c8dd95147474b9f91de67bc9c075c]) successfully deleted
I20240502 20:46:08.330394 4222 catalog_manager.cc:4830] TS ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193): tablet 789757df8e7d4284abcd7fcdf7538cf5 (table rebalance_test_table_10 [id=e54c8dd95147474b9f91de67bc9c075c]) successfully deleted
I20240502 20:46:08.330809 7297 ts_tablet_manager.cc:1875] T 9e34e8e17a7e413b97eb614a7df727d5 P c92e6e46782849479c8a0c6d82276fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:08.331230 7310 consensus_queue.cc:1035] T 747e7331d7ad4dd4b132672254037e1b P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:08.330883 4233 catalog_manager.cc:5484] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P ed47c023452a497192f35d2f617bf9bd reported cstate change: term changed from 0 to 1, leader changed from <none> to ed47c023452a497192f35d2f617bf9bd (127.30.32.68). New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:08.333601 7297 ts_tablet_manager.cc:1888] T 9e34e8e17a7e413b97eb614a7df727d5 P c92e6e46782849479c8a0c6d82276fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240502 20:46:08.334897 7299 ts_tablet_manager.cc:1888] T 9e34e8e17a7e413b97eb614a7df727d5 P ed47c023452a497192f35d2f617bf9bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240502 20:46:08.335300 7299 log.cc:1196] T 9e34e8e17a7e413b97eb614a7df727d5 P ed47c023452a497192f35d2f617bf9bd: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal/wals/9e34e8e17a7e413b97eb614a7df727d5
I20240502 20:46:08.335598 7299 ts_tablet_manager.cc:1909] T 9e34e8e17a7e413b97eb614a7df727d5 P ed47c023452a497192f35d2f617bf9bd: Deleting consensus metadata
I20240502 20:46:08.335968 4228 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41812:
name: "rebalancer_extra_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:46:08.336840 7297 log.cc:1196] T 9e34e8e17a7e413b97eb614a7df727d5 P c92e6e46782849479c8a0c6d82276fb2: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal/wals/9e34e8e17a7e413b97eb614a7df727d5
I20240502 20:46:08.337244 7297 ts_tablet_manager.cc:1909] T 9e34e8e17a7e413b97eb614a7df727d5 P c92e6e46782849479c8a0c6d82276fb2: Deleting consensus metadata
I20240502 20:46:08.337514 4222 catalog_manager.cc:4830] TS ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193): tablet 9e34e8e17a7e413b97eb614a7df727d5 (table rebalance_test_table_10 [id=e54c8dd95147474b9f91de67bc9c075c]) successfully deleted
I20240502 20:46:08.337908 4219 catalog_manager.cc:4830] TS 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): tablet 9e34e8e17a7e413b97eb614a7df727d5 (table rebalance_test_table_10 [id=e54c8dd95147474b9f91de67bc9c075c]) successfully deleted
I20240502 20:46:08.339058 7299 tablet_replica.cc:330] T f09dcfea3eb347e687f3c72f4ccc6929 P ed47c023452a497192f35d2f617bf9bd: stopping tablet replica
I20240502 20:46:08.339764 4222 catalog_manager.cc:4830] TS c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827): tablet 9e34e8e17a7e413b97eb614a7df727d5 (table rebalance_test_table_10 [id=e54c8dd95147474b9f91de67bc9c075c]) successfully deleted
I20240502 20:46:08.339877 6957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dc7d9f64ba2438fb37fbe0b33f4a8be" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47e5c56f13843018ac584914a8d4707" is_pre_election: true
I20240502 20:46:08.339991 6957 raft_consensus.cc:2454] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 0.
I20240502 20:46:08.340011 6967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dc7d9f64ba2438fb37fbe0b33f4a8be" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47e5c56f13843018ac584914a8d4707"
I20240502 20:46:08.340103 6967 raft_consensus.cc:3037] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:08.341046 6967 raft_consensus.cc:2454] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 1.
I20240502 20:46:08.341460 7299 raft_consensus.cc:2229] T f09dcfea3eb347e687f3c72f4ccc6929 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:08.341630 7299 raft_consensus.cc:2258] T f09dcfea3eb347e687f3c72f4ccc6929 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:08.342286 7299 ts_tablet_manager.cc:1875] T f09dcfea3eb347e687f3c72f4ccc6929 P ed47c023452a497192f35d2f617bf9bd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:08.344084 7299 ts_tablet_manager.cc:1888] T f09dcfea3eb347e687f3c72f4ccc6929 P ed47c023452a497192f35d2f617bf9bd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:46:08.344221 7299 log.cc:1196] T f09dcfea3eb347e687f3c72f4ccc6929 P ed47c023452a497192f35d2f617bf9bd: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal/wals/f09dcfea3eb347e687f3c72f4ccc6929
I20240502 20:46:08.345396 4222 catalog_manager.cc:4830] TS ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193): tablet f09dcfea3eb347e687f3c72f4ccc6929 (table rebalance_test_table_16 [id=dffd2e92869942b9a6f03505945ad3ba]) successfully deleted
I20240502 20:46:08.345533 4749 tablet_service.cc:1467] Processing CreateTablet for tablet 78d87e2d8c0b4bbd8dd6d7f49cdb4287 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=f1553e3c7d3c4023865edad13ab86f4a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:08.345832 4749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78d87e2d8c0b4bbd8dd6d7f49cdb4287. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:08.345957 6809 tablet_service.cc:1467] Processing CreateTablet for tablet 78d87e2d8c0b4bbd8dd6d7f49cdb4287 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=f1553e3c7d3c4023865edad13ab86f4a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:08.346247 6809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78d87e2d8c0b4bbd8dd6d7f49cdb4287. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:08.346581 4750 tablet_service.cc:1467] Processing CreateTablet for tablet 63f98288c8b1483493d5018c36e611bb (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=f1553e3c7d3c4023865edad13ab86f4a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:08.346642 7079 tablet_service.cc:1467] Processing CreateTablet for tablet 63f98288c8b1483493d5018c36e611bb (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=f1553e3c7d3c4023865edad13ab86f4a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:08.346850 4750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63f98288c8b1483493d5018c36e611bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:08.346904 7079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63f98288c8b1483493d5018c36e611bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:08.348225 6947 tablet_service.cc:1467] Processing CreateTablet for tablet 78d87e2d8c0b4bbd8dd6d7f49cdb4287 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=f1553e3c7d3c4023865edad13ab86f4a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:08.348353 8014 tablet_bootstrap.cc:492] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap starting.
I20240502 20:46:08.348501 6947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78d87e2d8c0b4bbd8dd6d7f49cdb4287. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:08.349416 6683 tablet_service.cc:1467] Processing CreateTablet for tablet 63f98288c8b1483493d5018c36e611bb (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=f1553e3c7d3c4023865edad13ab86f4a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:08.349560 8014 tablet_bootstrap.cc:654] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a4bf1df7fa6b4948be502c182bfbb833: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:08.349694 6683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63f98288c8b1483493d5018c36e611bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:08.350811 6946 tablet_service.cc:1467] Processing CreateTablet for tablet 8602022a45e1482ebf8dab9930ce8209 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=f1553e3c7d3c4023865edad13ab86f4a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:08.350941 6809 tablet_service.cc:1467] Processing CreateTablet for tablet 8602022a45e1482ebf8dab9930ce8209 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=f1553e3c7d3c4023865edad13ab86f4a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:08.351029 6946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8602022a45e1482ebf8dab9930ce8209. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:08.351135 6809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8602022a45e1482ebf8dab9930ce8209. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:08.352003 8308 tablet_bootstrap.cc:492] T 63f98288c8b1483493d5018c36e611bb P 7845c397230d49358ba2fad9db53f97c: Bootstrap starting.
I20240502 20:46:08.352161 6682 tablet_service.cc:1467] Processing CreateTablet for tablet 8602022a45e1482ebf8dab9930ce8209 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=f1553e3c7d3c4023865edad13ab86f4a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:08.352398 6682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8602022a45e1482ebf8dab9930ce8209. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:08.352998 8198 tablet_bootstrap.cc:492] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:46:08.353314 8308 tablet_bootstrap.cc:654] T 63f98288c8b1483493d5018c36e611bb P 7845c397230d49358ba2fad9db53f97c: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:08.353364 8014 tablet_bootstrap.cc:492] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a4bf1df7fa6b4948be502c182bfbb833: No bootstrap required, opened a new log
I20240502 20:46:08.353439 8014 ts_tablet_manager.cc:1367] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a4bf1df7fa6b4948be502c182bfbb833: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 20:46:08.353919 8198 tablet_bootstrap.cc:654] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:08.353902 8014 raft_consensus.cc:348] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:08.353992 8014 raft_consensus.cc:374] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:08.354014 8014 raft_consensus.cc:729] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Initialized, Role: FOLLOWER
I20240502 20:46:08.354125 8014 consensus_queue.cc:260] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a4bf1df7fa6b4948be502c182bfbb833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:08.354404 8014 ts_tablet_manager.cc:1398] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a4bf1df7fa6b4948be502c182bfbb833: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:08.354542 7939 tablet_bootstrap.cc:492] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a47e5c56f13843018ac584914a8d4707: Bootstrap starting.
I20240502 20:46:08.355152 8014 tablet_bootstrap.cc:492] T 8602022a45e1482ebf8dab9930ce8209 P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap starting.
I20240502 20:46:08.355499 7939 tablet_bootstrap.cc:654] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a47e5c56f13843018ac584914a8d4707: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:08.356257 8014 tablet_bootstrap.cc:654] T 8602022a45e1482ebf8dab9930ce8209 P a4bf1df7fa6b4948be502c182bfbb833: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:08.356379 8198 tablet_bootstrap.cc:492] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:46:08.356456 8198 ts_tablet_manager.cc:1367] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240502 20:46:08.356909 8198 raft_consensus.cc:348] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:08.357040 8198 raft_consensus.cc:374] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:08.357077 8198 raft_consensus.cc:729] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:46:08.357203 8198 consensus_queue.cc:260] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:08.357498 8198 ts_tablet_manager.cc:1398] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:08.358534 7939 tablet_bootstrap.cc:492] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a47e5c56f13843018ac584914a8d4707: No bootstrap required, opened a new log
I20240502 20:46:08.358635 7939 ts_tablet_manager.cc:1367] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a47e5c56f13843018ac584914a8d4707: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240502 20:46:08.358949 8198 tablet_bootstrap.cc:492] T 63f98288c8b1483493d5018c36e611bb P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:46:08.359095 7939 raft_consensus.cc:348] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:08.359210 7939 raft_consensus.cc:374] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:08.359248 7939 raft_consensus.cc:729] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Initialized, Role: FOLLOWER
I20240502 20:46:08.359360 7939 consensus_queue.cc:260] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a47e5c56f13843018ac584914a8d4707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:08.359650 7939 ts_tablet_manager.cc:1398] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a47e5c56f13843018ac584914a8d4707: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:08.359843 7939 tablet_bootstrap.cc:492] T 8602022a45e1482ebf8dab9930ce8209 P a47e5c56f13843018ac584914a8d4707: Bootstrap starting.
I20240502 20:46:08.358736 8014 tablet_bootstrap.cc:492] T 8602022a45e1482ebf8dab9930ce8209 P a4bf1df7fa6b4948be502c182bfbb833: No bootstrap required, opened a new log
I20240502 20:46:08.360186 8014 ts_tablet_manager.cc:1367] T 8602022a45e1482ebf8dab9930ce8209 P a4bf1df7fa6b4948be502c182bfbb833: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 20:46:08.360661 7939 tablet_bootstrap.cc:654] T 8602022a45e1482ebf8dab9930ce8209 P a47e5c56f13843018ac584914a8d4707: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:08.360647 8014 raft_consensus.cc:348] T 8602022a45e1482ebf8dab9930ce8209 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:08.360755 8014 raft_consensus.cc:374] T 8602022a45e1482ebf8dab9930ce8209 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:08.360810 8014 raft_consensus.cc:729] T 8602022a45e1482ebf8dab9930ce8209 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Initialized, Role: FOLLOWER
I20240502 20:46:08.360949 8014 consensus_queue.cc:260] T 8602022a45e1482ebf8dab9930ce8209 P a4bf1df7fa6b4948be502c182bfbb833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:08.361049 8198 tablet_bootstrap.cc:654] T 63f98288c8b1483493d5018c36e611bb P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:08.361831 8306 tablet_bootstrap.cc:492] T 63f98288c8b1483493d5018c36e611bb P 201d2b6485084e7dad2cbb722af94779: Bootstrap starting.
I20240502 20:46:08.361918 8014 ts_tablet_manager.cc:1398] T 8602022a45e1482ebf8dab9930ce8209 P a4bf1df7fa6b4948be502c182bfbb833: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:08.362713 7939 tablet_bootstrap.cc:492] T 8602022a45e1482ebf8dab9930ce8209 P a47e5c56f13843018ac584914a8d4707: No bootstrap required, opened a new log
I20240502 20:46:08.362808 7939 ts_tablet_manager.cc:1367] T 8602022a45e1482ebf8dab9930ce8209 P a47e5c56f13843018ac584914a8d4707: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 20:46:08.363214 7939 raft_consensus.cc:348] T 8602022a45e1482ebf8dab9930ce8209 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:08.363322 7939 raft_consensus.cc:374] T 8602022a45e1482ebf8dab9930ce8209 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:08.363335 8306 tablet_bootstrap.cc:654] T 63f98288c8b1483493d5018c36e611bb P 201d2b6485084e7dad2cbb722af94779: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:08.363358 7939 raft_consensus.cc:729] T 8602022a45e1482ebf8dab9930ce8209 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Initialized, Role: FOLLOWER
I20240502 20:46:08.363476 7939 consensus_queue.cc:260] T 8602022a45e1482ebf8dab9930ce8209 P a47e5c56f13843018ac584914a8d4707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:08.363750 7939 ts_tablet_manager.cc:1398] T 8602022a45e1482ebf8dab9930ce8209 P a47e5c56f13843018ac584914a8d4707: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:08.364655 8198 tablet_bootstrap.cc:492] T 63f98288c8b1483493d5018c36e611bb P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:46:08.364748 8198 ts_tablet_manager.cc:1367] T 63f98288c8b1483493d5018c36e611bb P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 20:46:08.365259 8198 raft_consensus.cc:348] T 63f98288c8b1483493d5018c36e611bb P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:08.365365 8198 raft_consensus.cc:374] T 63f98288c8b1483493d5018c36e611bb P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:08.365398 8198 raft_consensus.cc:729] T 63f98288c8b1483493d5018c36e611bb P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:46:08.365528 8198 consensus_queue.cc:260] T 63f98288c8b1483493d5018c36e611bb P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:08.365798 8198 ts_tablet_manager.cc:1398] T 63f98288c8b1483493d5018c36e611bb P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:08.366832 8306 tablet_bootstrap.cc:492] T 63f98288c8b1483493d5018c36e611bb P 201d2b6485084e7dad2cbb722af94779: No bootstrap required, opened a new log
I20240502 20:46:08.366943 8306 ts_tablet_manager.cc:1367] T 63f98288c8b1483493d5018c36e611bb P 201d2b6485084e7dad2cbb722af94779: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 20:46:08.367100 8308 tablet_bootstrap.cc:492] T 63f98288c8b1483493d5018c36e611bb P 7845c397230d49358ba2fad9db53f97c: No bootstrap required, opened a new log
I20240502 20:46:08.367244 8308 ts_tablet_manager.cc:1367] T 63f98288c8b1483493d5018c36e611bb P 7845c397230d49358ba2fad9db53f97c: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240502 20:46:08.367449 8306 raft_consensus.cc:348] T 63f98288c8b1483493d5018c36e611bb P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:08.367565 8306 raft_consensus.cc:374] T 63f98288c8b1483493d5018c36e611bb P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:08.367834 8306 raft_consensus.cc:729] T 63f98288c8b1483493d5018c36e611bb P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Initialized, Role: FOLLOWER
I20240502 20:46:08.367801 8308 raft_consensus.cc:348] T 63f98288c8b1483493d5018c36e611bb P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:08.367967 8308 raft_consensus.cc:374] T 63f98288c8b1483493d5018c36e611bb P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:08.368083 8308 raft_consensus.cc:729] T 63f98288c8b1483493d5018c36e611bb P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Initialized, Role: FOLLOWER
I20240502 20:46:08.368034 8306 consensus_queue.cc:260] T 63f98288c8b1483493d5018c36e611bb P 201d2b6485084e7dad2cbb722af94779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:08.368314 8308 consensus_queue.cc:260] T 63f98288c8b1483493d5018c36e611bb P 7845c397230d49358ba2fad9db53f97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:08.368876 8306 ts_tablet_manager.cc:1398] T 63f98288c8b1483493d5018c36e611bb P 201d2b6485084e7dad2cbb722af94779: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:08.369359 8308 ts_tablet_manager.cc:1398] T 63f98288c8b1483493d5018c36e611bb P 7845c397230d49358ba2fad9db53f97c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:08.369586 8308 tablet_bootstrap.cc:492] T 8602022a45e1482ebf8dab9930ce8209 P 7845c397230d49358ba2fad9db53f97c: Bootstrap starting.
I20240502 20:46:08.370348 8308 tablet_bootstrap.cc:654] T 8602022a45e1482ebf8dab9930ce8209 P 7845c397230d49358ba2fad9db53f97c: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:08.372071 8308 tablet_bootstrap.cc:492] T 8602022a45e1482ebf8dab9930ce8209 P 7845c397230d49358ba2fad9db53f97c: No bootstrap required, opened a new log
I20240502 20:46:08.372159 8308 ts_tablet_manager.cc:1367] T 8602022a45e1482ebf8dab9930ce8209 P 7845c397230d49358ba2fad9db53f97c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 20:46:08.372555 8308 raft_consensus.cc:348] T 8602022a45e1482ebf8dab9930ce8209 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:08.372660 8308 raft_consensus.cc:374] T 8602022a45e1482ebf8dab9930ce8209 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:08.372695 8308 raft_consensus.cc:729] T 8602022a45e1482ebf8dab9930ce8209 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Initialized, Role: FOLLOWER
I20240502 20:46:08.372834 8308 consensus_queue.cc:260] T 8602022a45e1482ebf8dab9930ce8209 P 7845c397230d49358ba2fad9db53f97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:08.373514 8308 ts_tablet_manager.cc:1398] T 8602022a45e1482ebf8dab9930ce8209 P 7845c397230d49358ba2fad9db53f97c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:08.389729 7996 raft_consensus.cc:482] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:08.389847 7996 raft_consensus.cc:504] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:08.390283 7996 leader_election.cc:288] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
I20240502 20:46:08.390756 6831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78d87e2d8c0b4bbd8dd6d7f49cdb4287" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833" is_pre_election: true
I20240502 20:46:08.390744 6967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78d87e2d8c0b4bbd8dd6d7f49cdb4287" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47e5c56f13843018ac584914a8d4707" is_pre_election: true
I20240502 20:46:08.390898 6831 raft_consensus.cc:2454] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 0.
I20240502 20:46:08.390904 6967 raft_consensus.cc:2454] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 0.
I20240502 20:46:08.391194 4704 leader_election.cc:302] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4bf1df7fa6b4948be502c182bfbb833, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:46:08.391398 7996 raft_consensus.cc:2781] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:08.391435 7996 raft_consensus.cc:482] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:08.391460 7996 raft_consensus.cc:3037] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:08.392396 7996 raft_consensus.cc:504] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:08.392815 7996 leader_election.cc:288] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 election: Requested vote from peers a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
I20240502 20:46:08.393659 6967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78d87e2d8c0b4bbd8dd6d7f49cdb4287" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47e5c56f13843018ac584914a8d4707"
I20240502 20:46:08.393787 6967 raft_consensus.cc:3037] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:08.393762 6831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78d87e2d8c0b4bbd8dd6d7f49cdb4287" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833"
I20240502 20:46:08.393877 6831 raft_consensus.cc:3037] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:08.394804 6831 raft_consensus.cc:2454] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 1.
I20240502 20:46:08.394804 6967 raft_consensus.cc:2454] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 1.
I20240502 20:46:08.395138 4706 leader_election.cc:302] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a47e5c56f13843018ac584914a8d4707, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:46:08.395397 7996 raft_consensus.cc:2781] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:08.395488 7996 raft_consensus.cc:686] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Becoming Leader. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Running, Role: LEADER
I20240502 20:46:08.395640 7996 consensus_queue.cc:237] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:08.397933 4233 catalog_manager.cc:5484] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd reported cstate change: term changed from 0 to 1, leader changed from <none> to ed47c023452a497192f35d2f617bf9bd (127.30.32.68). New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:08.398114 4506 consensus_queue.cc:237] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } }
I20240502 20:46:08.399714 4637 raft_consensus.cc:1261] T d6582234196c4fe4a82b3182787700de P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:08.400161 7967 consensus_queue.cc:1035] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:08.400581 4769 raft_consensus.cc:1261] T d6582234196c4fe4a82b3182787700de P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 20:46:08.400940 4443 consensus_peers.cc:485] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d6582234196c4fe4a82b3182787700de. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:08.401574 7967 consensus_queue.cc:1035] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:08.402684 7967 raft_consensus.cc:2932] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } } }
I20240502 20:46:08.404786 4769 raft_consensus.cc:2932] T d6582234196c4fe4a82b3182787700de P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } } }
I20240502 20:46:08.405074 4637 raft_consensus.cc:2932] T d6582234196c4fe4a82b3182787700de P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } } }
I20240502 20:46:08.405516 4233 catalog_manager.cc:5484] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: config changed from index -1 to 4, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:08.418558 4769 consensus_queue.cc:237] T a55d5b8b9ed04ca39888b89b82a3f833 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } }
I20240502 20:46:08.419960 4506 raft_consensus.cc:1261] T a55d5b8b9ed04ca39888b89b82a3f833 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:08.420114 4377 raft_consensus.cc:1261] T a55d5b8b9ed04ca39888b89b82a3f833 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 20:46:08.420348 4705 consensus_peers.cc:485] T a55d5b8b9ed04ca39888b89b82a3f833 P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a55d5b8b9ed04ca39888b89b82a3f833. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:08.420512 7996 consensus_queue.cc:1035] T a55d5b8b9ed04ca39888b89b82a3f833 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:08.420671 6241 consensus_queue.cc:1035] T a55d5b8b9ed04ca39888b89b82a3f833 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:08.422802 7318 raft_consensus.cc:2932] T a55d5b8b9ed04ca39888b89b82a3f833 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } } }
I20240502 20:46:08.423282 4377 raft_consensus.cc:2932] T a55d5b8b9ed04ca39888b89b82a3f833 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } } }
I20240502 20:46:08.425976 4229 catalog_manager.cc:5484] T a55d5b8b9ed04ca39888b89b82a3f833 P ed47c023452a497192f35d2f617bf9bd reported cstate change: config changed from index -1 to 4, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:08.430439 4506 raft_consensus.cc:2932] T a55d5b8b9ed04ca39888b89b82a3f833 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } } }
I20240502 20:46:08.436703 4506 consensus_queue.cc:237] T 5053a7afb4994ecdbc92b129c58903cf P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } }
I20240502 20:46:08.438191 4637 raft_consensus.cc:1261] T 5053a7afb4994ecdbc92b129c58903cf P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 20:46:08.438500 4442 consensus_peers.cc:485] T 5053a7afb4994ecdbc92b129c58903cf P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867): Couldn't send request to peer a4bf1df7fa6b4948be502c182bfbb833. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5053a7afb4994ecdbc92b129c58903cf. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:08.438508 4769 raft_consensus.cc:1261] T 5053a7afb4994ecdbc92b129c58903cf P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:08.438825 7921 consensus_queue.cc:1035] T 5053a7afb4994ecdbc92b129c58903cf P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:08.439064 6375 consensus_queue.cc:1035] T 5053a7afb4994ecdbc92b129c58903cf P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:08.440922 7967 raft_consensus.cc:2932] T 5053a7afb4994ecdbc92b129c58903cf P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } } }
I20240502 20:46:08.441393 4769 raft_consensus.cc:2932] T 5053a7afb4994ecdbc92b129c58903cf P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } } }
I20240502 20:46:08.441531 4637 raft_consensus.cc:2932] T 5053a7afb4994ecdbc92b129c58903cf P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } } }
I20240502 20:46:08.444257 4229 catalog_manager.cc:5484] T 5053a7afb4994ecdbc92b129c58903cf P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index -1 to 4, NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) added. New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } } }
I20240502 20:46:08.453706 6831 consensus_queue.cc:237] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } }
I20240502 20:46:08.455130 4769 raft_consensus.cc:1261] T 180902058815460189f75de4704d1e84 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer a4bf1df7fa6b4948be502c182bfbb833: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:08.455420 7099 raft_consensus.cc:1261] T 180902058815460189f75de4704d1e84 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Refusing update from remote peer a4bf1df7fa6b4948be502c182bfbb833: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 20:46:08.455507 6770 consensus_peers.cc:485] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 180902058815460189f75de4704d1e84. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:08.455924 7401 consensus_queue.cc:1035] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:46:08.456097 8364 consensus_queue.cc:1035] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:46:08.458097 7401 raft_consensus.cc:2932] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } } }
I20240502 20:46:08.458819 7099 raft_consensus.cc:2932] T 180902058815460189f75de4704d1e84 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } } }
I20240502 20:46:08.460595 4769 raft_consensus.cc:2932] T 180902058815460189f75de4704d1e84 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } } }
I20240502 20:46:08.461129 4234 catalog_manager.cc:5484] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833 reported cstate change: config changed from index -1 to 2, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New cstate: current_term: 1 leader_uuid: "a4bf1df7fa6b4948be502c182bfbb833" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:08.469820 4637 consensus_queue.cc:237] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } }
I20240502 20:46:08.471159 4769 raft_consensus.cc:1261] T 0cf7e95cb91b4e74b14365fcc6b7f968 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:08.471329 4377 raft_consensus.cc:1261] T 0cf7e95cb91b4e74b14365fcc6b7f968 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 20:46:08.471459 4575 consensus_peers.cc:485] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0cf7e95cb91b4e74b14365fcc6b7f968. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:08.471689 8168 consensus_queue.cc:1035] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:08.471884 8237 consensus_queue.cc:1035] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:08.473827 6613 raft_consensus.cc:2932] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } } }
I20240502 20:46:08.474361 4768 raft_consensus.cc:2932] T 0cf7e95cb91b4e74b14365fcc6b7f968 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } } }
I20240502 20:46:08.474361 4377 raft_consensus.cc:2932] T 0cf7e95cb91b4e74b14365fcc6b7f968 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } } }
I20240502 20:46:08.476683 4233 catalog_manager.cc:5484] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index -1 to 4, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:08.477716 4229 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 20:46:08.484191 6831 raft_consensus.cc:1261] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:08.484587 6967 raft_consensus.cc:1261] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:08.484617 8111 consensus_queue.cc:1035] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:08.485162 8111 consensus_queue.cc:1035] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:08.487130 4235 catalog_manager.cc:5576] TS a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867) does not have the latest schema for tablet 0dc7d9f64ba2438fb37fbe0b33f4a8be (table rebalancer_dynamic_table [id=4c3872020dc144bd8931994dced22724]). Expected version 1 got 0
I20240502 20:46:08.487375 4377 consensus_queue.cc:237] T 04c72d7d63574dea99cb2f7c14199a73 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } }
I20240502 20:46:08.488283 4235 catalog_manager.cc:5576] TS a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841) does not have the latest schema for tablet 0dc7d9f64ba2438fb37fbe0b33f4a8be (table rebalancer_dynamic_table [id=4c3872020dc144bd8931994dced22724]). Expected version 1 got 0
I20240502 20:46:08.490589 4506 raft_consensus.cc:1261] T 04c72d7d63574dea99cb2f7c14199a73 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:08.490672 4768 raft_consensus.cc:1261] T 04c72d7d63574dea99cb2f7c14199a73 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:08.491123 6138 consensus_queue.cc:1035] T 04c72d7d63574dea99cb2f7c14199a73 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:08.491405 4229 catalog_manager.cc:5576] TS ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193) does not have the latest schema for tablet 0dc7d9f64ba2438fb37fbe0b33f4a8be (table rebalancer_dynamic_table [id=4c3872020dc144bd8931994dced22724]). Expected version 1 got 0
I20240502 20:46:08.491533 6138 consensus_queue.cc:1035] T 04c72d7d63574dea99cb2f7c14199a73 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:08.491829 8383 tablet.cc:1722] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P ed47c023452a497192f35d2f617bf9bd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 20:46:08.493258 8390 tablet.cc:1722] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P a4bf1df7fa6b4948be502c182bfbb833: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 20:46:08.494056 8390 tablet.cc:1620] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P a4bf1df7fa6b4948be502c182bfbb833: MemRowSet was empty: no flush needed.
I20240502 20:46:08.494450 4228 catalog_manager.cc:5576] TS a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867) does not have the latest schema for tablet 0dc7d9f64ba2438fb37fbe0b33f4a8be (table rebalancer_dynamic_table [id=4c3872020dc144bd8931994dced22724]). Expected version 1 got 0
I20240502 20:46:08.494691 8383 tablet.cc:1620] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P ed47c023452a497192f35d2f617bf9bd: MemRowSet was empty: no flush needed.
I20240502 20:46:08.495041 4233 catalog_manager.cc:5576] TS a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841) does not have the latest schema for tablet 0dc7d9f64ba2438fb37fbe0b33f4a8be (table rebalancer_dynamic_table [id=4c3872020dc144bd8931994dced22724]). Expected version 1 got 0
I20240502 20:46:08.495882 8385 tablet.cc:1722] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P a47e5c56f13843018ac584914a8d4707: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 20:46:08.495972 4235 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: rebalancer_dynamic_table [id=4c3872020dc144bd8931994dced22724] alter complete (version 1)
I20240502 20:46:08.496214 8385 tablet.cc:1620] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P a47e5c56f13843018ac584914a8d4707: MemRowSet was empty: no flush needed.
I20240502 20:46:08.497442 8181 raft_consensus.cc:2932] T 04c72d7d63574dea99cb2f7c14199a73 P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } } }
I20240502 20:46:08.500113 4768 raft_consensus.cc:2932] T 04c72d7d63574dea99cb2f7c14199a73 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } } }
I20240502 20:46:08.500811 4235 catalog_manager.cc:5484] T 04c72d7d63574dea99cb2f7c14199a73 P c92e6e46782849479c8a0c6d82276fb2 reported cstate change: config changed from index -1 to 4, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New cstate: current_term: 1 leader_uuid: "c92e6e46782849479c8a0c6d82276fb2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 20:46:08.505528 4312 consensus_peers.cc:485] T 04c72d7d63574dea99cb2f7c14199a73 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 04c72d7d63574dea99cb2f7c14199a73. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:08.506256 4506 raft_consensus.cc:2932] T 04c72d7d63574dea99cb2f7c14199a73 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } } }
I20240502 20:46:08.511406 4637 consensus_queue.cc:237] T ed6386b6777c4d50b29ff8cd49894eba P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } }
I20240502 20:46:08.512741 4768 raft_consensus.cc:1261] T ed6386b6777c4d50b29ff8cd49894eba P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:08.512972 4506 raft_consensus.cc:1261] T ed6386b6777c4d50b29ff8cd49894eba P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:08.513428 6613 consensus_queue.cc:1035] T ed6386b6777c4d50b29ff8cd49894eba P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 20:46:08.513610 4573 consensus_peers.cc:485] T ed6386b6777c4d50b29ff8cd49894eba P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867): Couldn't send request to peer a4bf1df7fa6b4948be502c182bfbb833. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ed6386b6777c4d50b29ff8cd49894eba. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:08.513624 8236 consensus_queue.cc:1035] T ed6386b6777c4d50b29ff8cd49894eba P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:08.515578 7017 raft_consensus.cc:2932] T ed6386b6777c4d50b29ff8cd49894eba P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } } }
I20240502 20:46:08.515964 4768 raft_consensus.cc:2932] T ed6386b6777c4d50b29ff8cd49894eba P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } } }
I20240502 20:46:08.516103 4506 raft_consensus.cc:2932] T ed6386b6777c4d50b29ff8cd49894eba P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } } }
I20240502 20:46:08.518848 4233 catalog_manager.cc:5484] T ed6386b6777c4d50b29ff8cd49894eba P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index -1 to 4, NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) added. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:08.519929 8304 raft_consensus.cc:482] T 8602022a45e1482ebf8dab9930ce8209 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:08.520047 8304 raft_consensus.cc:504] T 8602022a45e1482ebf8dab9930ce8209 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:08.520496 8304 leader_election.cc:288] T 8602022a45e1482ebf8dab9930ce8209 P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867), 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
I20240502 20:46:08.520829 6831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8602022a45e1482ebf8dab9930ce8209" candidate_uuid: "a47e5c56f13843018ac584914a8d4707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833" is_pre_election: true
I20240502 20:46:08.520965 6831 raft_consensus.cc:2454] T 8602022a45e1482ebf8dab9930ce8209 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a47e5c56f13843018ac584914a8d4707 in term 0.
I20240502 20:46:08.521281 6901 leader_election.cc:302] T 8602022a45e1482ebf8dab9930ce8209 P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a47e5c56f13843018ac584914a8d4707, a4bf1df7fa6b4948be502c182bfbb833; no voters:
I20240502 20:46:08.521381 6702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8602022a45e1482ebf8dab9930ce8209" candidate_uuid: "a47e5c56f13843018ac584914a8d4707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7845c397230d49358ba2fad9db53f97c" is_pre_election: true
I20240502 20:46:08.521450 8304 raft_consensus.cc:2781] T 8602022a45e1482ebf8dab9930ce8209 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:08.521509 8304 raft_consensus.cc:482] T 8602022a45e1482ebf8dab9930ce8209 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:08.521513 6702 raft_consensus.cc:2454] T 8602022a45e1482ebf8dab9930ce8209 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a47e5c56f13843018ac584914a8d4707 in term 0.
I20240502 20:46:08.521553 8304 raft_consensus.cc:3037] T 8602022a45e1482ebf8dab9930ce8209 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:08.522549 8304 raft_consensus.cc:504] T 8602022a45e1482ebf8dab9930ce8209 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:08.522980 8304 leader_election.cc:288] T 8602022a45e1482ebf8dab9930ce8209 P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 1 election: Requested vote from peers a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867), 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
I20240502 20:46:08.523423 6831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8602022a45e1482ebf8dab9930ce8209" candidate_uuid: "a47e5c56f13843018ac584914a8d4707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833"
I20240502 20:46:08.523571 6831 raft_consensus.cc:3037] T 8602022a45e1482ebf8dab9930ce8209 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:08.523669 6702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8602022a45e1482ebf8dab9930ce8209" candidate_uuid: "a47e5c56f13843018ac584914a8d4707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7845c397230d49358ba2fad9db53f97c"
I20240502 20:46:08.523793 6702 raft_consensus.cc:3037] T 8602022a45e1482ebf8dab9930ce8209 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:08.524377 6831 raft_consensus.cc:2454] T 8602022a45e1482ebf8dab9930ce8209 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a47e5c56f13843018ac584914a8d4707 in term 1.
I20240502 20:46:08.524487 6702 raft_consensus.cc:2454] T 8602022a45e1482ebf8dab9930ce8209 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a47e5c56f13843018ac584914a8d4707 in term 1.
I20240502 20:46:08.524644 6901 leader_election.cc:302] T 8602022a45e1482ebf8dab9930ce8209 P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a47e5c56f13843018ac584914a8d4707, a4bf1df7fa6b4948be502c182bfbb833; no voters:
I20240502 20:46:08.524968 8304 raft_consensus.cc:2781] T 8602022a45e1482ebf8dab9930ce8209 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:08.525087 8304 raft_consensus.cc:686] T 8602022a45e1482ebf8dab9930ce8209 P a47e5c56f13843018ac584914a8d4707 [term 1 LEADER]: Becoming Leader. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Running, Role: LEADER
I20240502 20:46:08.525209 8304 consensus_queue.cc:237] T 8602022a45e1482ebf8dab9930ce8209 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:08.526803 4235 catalog_manager.cc:5484] T 8602022a45e1482ebf8dab9930ce8209 P a47e5c56f13843018ac584914a8d4707 reported cstate change: term changed from 0 to 1, leader changed from <none> to a47e5c56f13843018ac584914a8d4707 (127.30.32.71). New cstate: current_term: 1 leader_uuid: "a47e5c56f13843018ac584914a8d4707" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:08.531191 4768 consensus_queue.cc:237] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } }
I20240502 20:46:08.532676 4637 raft_consensus.cc:1261] T cdf218d5e45b4db19a8e2c22dbc5cc5d P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:08.532852 4377 raft_consensus.cc:1261] T cdf218d5e45b4db19a8e2c22dbc5cc5d P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:08.533264 8111 consensus_queue.cc:1035] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:08.534888 7996 consensus_queue.cc:1035] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 20:46:08.535065 4703 consensus_peers.cc:485] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cdf218d5e45b4db19a8e2c22dbc5cc5d. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:08.535053 7318 raft_consensus.cc:2932] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } } }
I20240502 20:46:08.535967 4637 raft_consensus.cc:2932] T cdf218d5e45b4db19a8e2c22dbc5cc5d P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } } }
I20240502 20:46:08.537196 4377 raft_consensus.cc:2932] T cdf218d5e45b4db19a8e2c22dbc5cc5d P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } } }
I20240502 20:46:08.538694 4228 catalog_manager.cc:5484] T cdf218d5e45b4db19a8e2c22dbc5cc5d P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index -1 to 4, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } } }
I20240502 20:46:08.545069 6375 raft_consensus.cc:979] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc: : Instructing follower c92e6e46782849479c8a0c6d82276fb2 to start an election
I20240502 20:46:08.545181 6375 raft_consensus.cc:1067] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Signalling peer c92e6e46782849479c8a0c6d82276fb2 to start an election
I20240502 20:46:08.545734 4377 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5a36bf6e8e6e43b2b47b6d6357b0d445"
dest_uuid: "c92e6e46782849479c8a0c6d82276fb2"
from {username='slave'} at 127.30.32.66:60785
I20240502 20:46:08.545887 4377 raft_consensus.cc:482] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:46:08.545969 4377 raft_consensus.cc:3037] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:08.546789 4377 raft_consensus.cc:504] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c92e6e46782849479c8a0c6d82276fb2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } }
I20240502 20:46:08.547215 4377 leader_election.cc:288] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 election: Requested vote from peers c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097), 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
I20240502 20:46:08.547189 4768 consensus_queue.cc:237] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } }
I20240502 20:46:08.547441 4506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a36bf6e8e6e43b2b47b6d6357b0d445" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc"
I20240502 20:46:08.547554 4506 raft_consensus.cc:3032] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:46:08.547592 4506 raft_consensus.cc:729] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Running, Role: LEADER
I20240502 20:46:08.547758 4506 consensus_queue.cc:260] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } }
I20240502 20:46:08.547799 4637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a36bf6e8e6e43b2b47b6d6357b0d445" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91"
I20240502 20:46:08.547895 4506 raft_consensus.cc:3037] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:08.547923 4637 raft_consensus.cc:3037] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:08.548835 4506 raft_consensus.cc:2454] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 2.
I20240502 20:46:08.548949 4507 raft_consensus.cc:1261] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 20:46:08.549160 4706 consensus_peers.cc:485] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P ed47c023452a497192f35d2f617bf9bd -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6faa71d4bd2143b0a72cd86a7e1e7e12. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:08.549147 7099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a36bf6e8e6e43b2b47b6d6357b0d445" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "201d2b6485084e7dad2cbb722af94779"
I20240502 20:46:08.549275 4377 raft_consensus.cc:1261] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:08.549274 7099 raft_consensus.cc:3037] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:08.549399 7996 consensus_queue.cc:1035] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:08.550269 7996 consensus_queue.cc:1035] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:08.550341 7099 raft_consensus.cc:2454] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 201d2b6485084e7dad2cbb722af94779 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 2.
I20240502 20:46:08.548970 4637 raft_consensus.cc:2454] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 2.
I20240502 20:46:08.550668 4310 leader_election.cc:302] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 201d2b6485084e7dad2cbb722af94779, c3d26eafd51547acb780d4aac5ebe7cc, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:46:08.551033 8185 raft_consensus.cc:2781] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c92e6e46782849479c8a0c6d82276fb2 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:46:08.551118 8185 raft_consensus.cc:686] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c92e6e46782849479c8a0c6d82276fb2 [term 2 LEADER]: Becoming Leader. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Running, Role: LEADER
I20240502 20:46:08.551257 8185 consensus_queue.cc:237] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } }
I20240502 20:46:08.552002 8111 raft_consensus.cc:2932] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } } }
I20240502 20:46:08.553602 4229 catalog_manager.cc:5484] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c92e6e46782849479c8a0c6d82276fb2 reported cstate change: term changed from 1 to 2, leader changed from c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) to c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65). New cstate: current_term: 2 leader_uuid: "c92e6e46782849479c8a0c6d82276fb2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:08.554677 4506 raft_consensus.cc:2932] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } } }
I20240502 20:46:08.555039 4235 catalog_manager.cc:5484] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P ed47c023452a497192f35d2f617bf9bd reported cstate change: config changed from index -1 to 4, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:08.557575 4377 raft_consensus.cc:2932] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } } }
I20240502 20:46:08.564872 7318 raft_consensus.cc:482] T 63f98288c8b1483493d5018c36e611bb P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:08.564965 7318 raft_consensus.cc:504] T 63f98288c8b1483493d5018c36e611bb P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:08.565327 7318 leader_election.cc:288] T 63f98288c8b1483493d5018c36e611bb P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
I20240502 20:46:08.565675 7099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f98288c8b1483493d5018c36e611bb" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201d2b6485084e7dad2cbb722af94779" is_pre_election: true
I20240502 20:46:08.565711 6702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f98288c8b1483493d5018c36e611bb" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7845c397230d49358ba2fad9db53f97c" is_pre_election: true
I20240502 20:46:08.565810 7099 raft_consensus.cc:2454] T 63f98288c8b1483493d5018c36e611bb P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 0.
I20240502 20:46:08.565843 6702 raft_consensus.cc:2454] T 63f98288c8b1483493d5018c36e611bb P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 0.
I20240502 20:46:08.566076 4703 leader_election.cc:302] T 63f98288c8b1483493d5018c36e611bb P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201d2b6485084e7dad2cbb722af94779, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:46:08.566246 7318 raft_consensus.cc:2781] T 63f98288c8b1483493d5018c36e611bb P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:08.566304 7318 raft_consensus.cc:482] T 63f98288c8b1483493d5018c36e611bb P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:08.566358 7318 raft_consensus.cc:3037] T 63f98288c8b1483493d5018c36e611bb P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:08.567162 7318 raft_consensus.cc:504] T 63f98288c8b1483493d5018c36e611bb P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:08.567493 7318 leader_election.cc:288] T 63f98288c8b1483493d5018c36e611bb P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 election: Requested vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
I20240502 20:46:08.567993 6702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f98288c8b1483493d5018c36e611bb" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7845c397230d49358ba2fad9db53f97c"
I20240502 20:46:08.568106 6702 raft_consensus.cc:3037] T 63f98288c8b1483493d5018c36e611bb P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:08.568284 7099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f98288c8b1483493d5018c36e611bb" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201d2b6485084e7dad2cbb722af94779"
I20240502 20:46:08.568382 7099 raft_consensus.cc:3037] T 63f98288c8b1483493d5018c36e611bb P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:08.569013 6702 raft_consensus.cc:2454] T 63f98288c8b1483493d5018c36e611bb P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 1.
I20240502 20:46:08.569175 7099 raft_consensus.cc:2454] T 63f98288c8b1483493d5018c36e611bb P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 1.
I20240502 20:46:08.569607 4705 leader_election.cc:302] T 63f98288c8b1483493d5018c36e611bb P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7845c397230d49358ba2fad9db53f97c, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:46:08.569774 6241 raft_consensus.cc:2781] T 63f98288c8b1483493d5018c36e611bb P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:08.569847 6241 raft_consensus.cc:686] T 63f98288c8b1483493d5018c36e611bb P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Becoming Leader. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Running, Role: LEADER
I20240502 20:46:08.569962 6241 consensus_queue.cc:237] T 63f98288c8b1483493d5018c36e611bb P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:08.571826 4229 catalog_manager.cc:5484] T 63f98288c8b1483493d5018c36e611bb P ed47c023452a497192f35d2f617bf9bd reported cstate change: term changed from 0 to 1, leader changed from <none> to ed47c023452a497192f35d2f617bf9bd (127.30.32.68). New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:08.601368 4226 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 20:46:08.607091 8383 tablet.cc:1722] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P ed47c023452a497192f35d2f617bf9bd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 20:46:08.607429 8383 tablet.cc:1620] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P ed47c023452a497192f35d2f617bf9bd: MemRowSet was empty: no flush needed.
I20240502 20:46:08.608317 8385 tablet.cc:1722] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P a47e5c56f13843018ac584914a8d4707: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 20:46:08.608618 8385 tablet.cc:1620] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P a47e5c56f13843018ac584914a8d4707: MemRowSet was empty: no flush needed.
I20240502 20:46:08.614413 8390 tablet.cc:1722] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P a4bf1df7fa6b4948be502c182bfbb833: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 20:46:08.614714 8390 tablet.cc:1620] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P a4bf1df7fa6b4948be502c182bfbb833: MemRowSet was empty: no flush needed.
I20240502 20:46:08.614925 4235 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: rebalancer_dynamic_table [id=4c3872020dc144bd8931994dced22724] alter complete (version 2)
I20240502 20:46:08.619725 8453 ts_tablet_manager.cc:902] T 66b8f583dfd24c2aa5a59457a8e58fbb P a47e5c56f13843018ac584914a8d4707: Initiating tablet copy from peer c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097)
I20240502 20:46:08.620368 8453 tablet_copy_client.cc:308] T 66b8f583dfd24c2aa5a59457a8e58fbb P a47e5c56f13843018ac584914a8d4707: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.66:42097
I20240502 20:46:08.620929 4528 tablet_copy_service.cc:140] P c3d26eafd51547acb780d4aac5ebe7cc: Received BeginTabletCopySession request for tablet 66b8f583dfd24c2aa5a59457a8e58fbb from peer a47e5c56f13843018ac584914a8d4707 ({username='slave'} at 127.30.32.71:46593)
I20240502 20:46:08.621091 4528 tablet_copy_service.cc:161] P c3d26eafd51547acb780d4aac5ebe7cc: Beginning new tablet copy session on tablet 66b8f583dfd24c2aa5a59457a8e58fbb from peer a47e5c56f13843018ac584914a8d4707 at {username='slave'} at 127.30.32.71:46593: session id = a47e5c56f13843018ac584914a8d4707-66b8f583dfd24c2aa5a59457a8e58fbb
I20240502 20:46:08.622016 4528 tablet_copy_source_session.cc:205] T 66b8f583dfd24c2aa5a59457a8e58fbb P c3d26eafd51547acb780d4aac5ebe7cc: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:08.622891 8453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66b8f583dfd24c2aa5a59457a8e58fbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:08.626104 8453 tablet_copy_client.cc:783] T 66b8f583dfd24c2aa5a59457a8e58fbb P a47e5c56f13843018ac584914a8d4707: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:08.626269 8453 tablet_copy_client.cc:647] T 66b8f583dfd24c2aa5a59457a8e58fbb P a47e5c56f13843018ac584914a8d4707: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:08.627437 8453 tablet_copy_client.cc:523] T 66b8f583dfd24c2aa5a59457a8e58fbb P a47e5c56f13843018ac584914a8d4707: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:08.628435 8453 tablet_bootstrap.cc:492] T 66b8f583dfd24c2aa5a59457a8e58fbb P a47e5c56f13843018ac584914a8d4707: Bootstrap starting.
I20240502 20:46:08.661976 8453 tablet_bootstrap.cc:492] T 66b8f583dfd24c2aa5a59457a8e58fbb P a47e5c56f13843018ac584914a8d4707: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:08.662475 8453 tablet_bootstrap.cc:492] T 66b8f583dfd24c2aa5a59457a8e58fbb P a47e5c56f13843018ac584914a8d4707: Bootstrap complete.
I20240502 20:46:08.662617 6682 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:08.662773 8453 ts_tablet_manager.cc:1367] T 66b8f583dfd24c2aa5a59457a8e58fbb P a47e5c56f13843018ac584914a8d4707: Time spent bootstrapping tablet: real 0.034s user 0.029s sys 0.000s
I20240502 20:46:08.663336 8453 raft_consensus.cc:348] T 66b8f583dfd24c2aa5a59457a8e58fbb P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } }
I20240502 20:46:08.664222 8453 raft_consensus.cc:729] T 66b8f583dfd24c2aa5a59457a8e58fbb P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Initialized, Role: LEARNER
I20240502 20:46:08.664368 8453 consensus_queue.cc:260] T 66b8f583dfd24c2aa5a59457a8e58fbb P a47e5c56f13843018ac584914a8d4707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } }
I20240502 20:46:08.664754 8453 ts_tablet_manager.cc:1398] T 66b8f583dfd24c2aa5a59457a8e58fbb P a47e5c56f13843018ac584914a8d4707: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:08.666066 4528 tablet_copy_service.cc:342] P c3d26eafd51547acb780d4aac5ebe7cc: Request end of tablet copy session a47e5c56f13843018ac584914a8d4707-66b8f583dfd24c2aa5a59457a8e58fbb received from {username='slave'} at 127.30.32.71:46593
I20240502 20:46:08.666153 4528 tablet_copy_service.cc:433] P c3d26eafd51547acb780d4aac5ebe7cc: ending tablet copy session a47e5c56f13843018ac584914a8d4707-66b8f583dfd24c2aa5a59457a8e58fbb on tablet 66b8f583dfd24c2aa5a59457a8e58fbb with peer a47e5c56f13843018ac584914a8d4707
I20240502 20:46:08.667212 7079 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:08.677615 6946 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:08.678356 6809 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:08.679747 4487 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 20:46:08.689373 4618 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 20:46:08.693142 4750 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 20:46:08.699678 4354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:46:08.719280 4227 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 20:46:08.729761 8383 tablet.cc:1722] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P ed47c023452a497192f35d2f617bf9bd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 20:46:08.730170 8385 tablet.cc:1722] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P a47e5c56f13843018ac584914a8d4707: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 20:46:08.730589 8390 tablet.cc:1722] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P a4bf1df7fa6b4948be502c182bfbb833: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 20:46:08.736588 4227 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: rebalancer_dynamic_table_new_name [id=4c3872020dc144bd8931994dced22724] alter complete (version 3)
I20240502 20:46:08.763604 4768 consensus_queue.cc:237] T e5a4dc24914c4f37a14481b77c4fcdaa P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } }
I20240502 20:46:08.764909 4506 raft_consensus.cc:1261] T e5a4dc24914c4f37a14481b77c4fcdaa P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 20:46:08.765102 4703 consensus_peers.cc:485] T e5a4dc24914c4f37a14481b77c4fcdaa P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e5a4dc24914c4f37a14481b77c4fcdaa. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:08.765337 4377 raft_consensus.cc:1261] T e5a4dc24914c4f37a14481b77c4fcdaa P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:08.765728 8111 consensus_queue.cc:1035] T e5a4dc24914c4f37a14481b77c4fcdaa P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:08.765900 6241 consensus_queue.cc:1035] T e5a4dc24914c4f37a14481b77c4fcdaa P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:08.767707 7318 raft_consensus.cc:2932] T e5a4dc24914c4f37a14481b77c4fcdaa P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } } }
I20240502 20:46:08.768146 4377 raft_consensus.cc:2932] T e5a4dc24914c4f37a14481b77c4fcdaa P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } } }
I20240502 20:46:08.769337 4506 raft_consensus.cc:2932] T e5a4dc24914c4f37a14481b77c4fcdaa P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } } }
I20240502 20:46:08.770401 4233 catalog_manager.cc:5484] T e5a4dc24914c4f37a14481b77c4fcdaa P ed47c023452a497192f35d2f617bf9bd reported cstate change: config changed from index -1 to 4, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:08.779841 4506 consensus_queue.cc:237] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } }
I20240502 20:46:08.781091 4768 raft_consensus.cc:1261] T 45ade8c672d8445eaffb179772e3759a P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:08.781116 4637 raft_consensus.cc:1261] T 45ade8c672d8445eaffb179772e3759a P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 20:46:08.781270 4442 consensus_peers.cc:485] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867): Couldn't send request to peer a4bf1df7fa6b4948be502c182bfbb833. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 45ade8c672d8445eaffb179772e3759a. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:08.781579 6375 consensus_queue.cc:1035] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:08.781740 7921 consensus_queue.cc:1035] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:08.782259 8111 consensus_queue.cc:1035] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:08.783650 7921 raft_consensus.cc:2932] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } } }
I20240502 20:46:08.784440 4768 raft_consensus.cc:2932] T 45ade8c672d8445eaffb179772e3759a P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } } }
I20240502 20:46:08.786947 4637 raft_consensus.cc:2932] T 45ade8c672d8445eaffb179772e3759a P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } } }
I20240502 20:46:08.787402 4227 catalog_manager.cc:5484] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: config changed from index -1 to 4, NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) added. New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:08.787617 7318 consensus_queue.cc:1035] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:08.798525 4637 consensus_queue.cc:237] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } }
I20240502 20:46:08.799885 4768 raft_consensus.cc:1261] T eebddad3efe748f78372901ecad5b407 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:08.800253 4374 raft_consensus.cc:1261] T eebddad3efe748f78372901ecad5b407 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:08.800263 8236 consensus_queue.cc:1035] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:08.800603 7017 consensus_queue.cc:1035] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:08.802799 7017 raft_consensus.cc:2932] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } } }
I20240502 20:46:08.803246 4374 raft_consensus.cc:2932] T eebddad3efe748f78372901ecad5b407 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } } }
I20240502 20:46:08.804412 4768 raft_consensus.cc:2932] T eebddad3efe748f78372901ecad5b407 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } } }
W20240502 20:46:08.805893 4572 consensus_peers.cc:485] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eebddad3efe748f78372901ecad5b407. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:08.805788 4227 catalog_manager.cc:5484] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index -1 to 4, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:08.815640 4374 consensus_queue.cc:237] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } }
I20240502 20:46:08.816408 4228 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240502 20:46:08.816548 4228 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240502 20:46:08.817322 4507 raft_consensus.cc:1261] T c55ba788f592448d8941ff94a8cefc52 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 20:46:08.817618 4313 consensus_peers.cc:485] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c55ba788f592448d8941ff94a8cefc52. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:08.817970 6138 consensus_queue.cc:1035] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:08.818293 4637 raft_consensus.cc:1261] T c55ba788f592448d8941ff94a8cefc52 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:08.818729 6138 consensus_queue.cc:1035] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:08.819257 4228 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: Sending DeleteTablet for 3 replicas of tablet be40862863f44123916c80a1ae5b24a9
I20240502 20:46:08.819607 4228 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: Sending DeleteTablet for 4 replicas of tablet 6faa71d4bd2143b0a72cd86a7e1e7e12
I20240502 20:46:08.819752 4487 tablet_service.cc:1514] Processing DeleteTablet for tablet be40862863f44123916c80a1ae5b24a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:08 UTC) from {username='slave'} at 127.0.0.1:48890
I20240502 20:46:08.819921 8141 tablet_replica.cc:330] T be40862863f44123916c80a1ae5b24a9 P c3d26eafd51547acb780d4aac5ebe7cc: stopping tablet replica
I20240502 20:46:08.819967 4228 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: Sending DeleteTablet for 3 replicas of tablet f352efa74d3c4da3907efdc37237211f
I20240502 20:46:08.820055 8141 raft_consensus.cc:2229] T be40862863f44123916c80a1ae5b24a9 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:08.820039 4354 tablet_service.cc:1514] Processing DeleteTablet for tablet be40862863f44123916c80a1ae5b24a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:08 UTC) from {username='slave'} at 127.0.0.1:38306
I20240502 20:46:08.820183 8141 raft_consensus.cc:2258] T be40862863f44123916c80a1ae5b24a9 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:08.820384 4618 tablet_service.cc:1514] Processing DeleteTablet for tablet be40862863f44123916c80a1ae5b24a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:08 UTC) from {username='slave'} at 127.0.0.1:57068
I20240502 20:46:08.820552 7079 tablet_service.cc:1514] Processing DeleteTablet for tablet f352efa74d3c4da3907efdc37237211f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:08 UTC) from {username='slave'} at 127.0.0.1:59162
I20240502 20:46:08.820744 8137 tablet_replica.cc:330] T f352efa74d3c4da3907efdc37237211f P 201d2b6485084e7dad2cbb722af94779: stopping tablet replica
I20240502 20:46:08.820794 8141 ts_tablet_manager.cc:1875] T be40862863f44123916c80a1ae5b24a9 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:08.820895 8137 raft_consensus.cc:2229] T f352efa74d3c4da3907efdc37237211f P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:08.821060 8137 raft_consensus.cc:2258] T f352efa74d3c4da3907efdc37237211f P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:08.821074 7297 tablet_replica.cc:330] T be40862863f44123916c80a1ae5b24a9 P c92e6e46782849479c8a0c6d82276fb2: stopping tablet replica
I20240502 20:46:08.821235 7297 raft_consensus.cc:2229] T be40862863f44123916c80a1ae5b24a9 P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:08.821239 4749 tablet_service.cc:1514] Processing DeleteTablet for tablet f352efa74d3c4da3907efdc37237211f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:08 UTC) from {username='slave'} at 127.0.0.1:38526
I20240502 20:46:08.821322 4354 tablet_service.cc:1514] Processing DeleteTablet for tablet 6faa71d4bd2143b0a72cd86a7e1e7e12 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:08 UTC) from {username='slave'} at 127.0.0.1:38306
I20240502 20:46:08.821422 7297 raft_consensus.cc:2258] T be40862863f44123916c80a1ae5b24a9 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:08.821624 4618 tablet_service.cc:1514] Processing DeleteTablet for tablet f352efa74d3c4da3907efdc37237211f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:08 UTC) from {username='slave'} at 127.0.0.1:57068
I20240502 20:46:08.821794 6138 raft_consensus.cc:2932] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } } }
I20240502 20:46:08.821925 7297 ts_tablet_manager.cc:1875] T be40862863f44123916c80a1ae5b24a9 P c92e6e46782849479c8a0c6d82276fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:08.821959 8137 ts_tablet_manager.cc:1875] T f352efa74d3c4da3907efdc37237211f P 201d2b6485084e7dad2cbb722af94779: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:08.823370 6946 tablet_service.cc:1514] Processing DeleteTablet for tablet 6faa71d4bd2143b0a72cd86a7e1e7e12 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:08 UTC) from {username='slave'} at 127.0.0.1:45156
I20240502 20:46:08.823773 7297 ts_tablet_manager.cc:1888] T be40862863f44123916c80a1ae5b24a9 P c92e6e46782849479c8a0c6d82276fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 20:46:08.823840 7297 log.cc:1196] T be40862863f44123916c80a1ae5b24a9 P c92e6e46782849479c8a0c6d82276fb2: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal/wals/be40862863f44123916c80a1ae5b24a9
W20240502 20:46:08.824066 4222 catalog_manager.cc:4797] TS a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): delete failed for tablet 6faa71d4bd2143b0a72cd86a7e1e7e12 because the tablet was not found. No further retry: Not found: Tablet not found: 6faa71d4bd2143b0a72cd86a7e1e7e12
I20240502 20:46:08.824117 7297 ts_tablet_manager.cc:1909] T be40862863f44123916c80a1ae5b24a9 P c92e6e46782849479c8a0c6d82276fb2: Deleting consensus metadata
I20240502 20:46:08.824115 8137 ts_tablet_manager.cc:1888] T f352efa74d3c4da3907efdc37237211f P 201d2b6485084e7dad2cbb722af94779: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240502 20:46:08.824182 8137 log.cc:1196] T f352efa74d3c4da3907efdc37237211f P 201d2b6485084e7dad2cbb722af94779: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/wal/wals/f352efa74d3c4da3907efdc37237211f
I20240502 20:46:08.824430 8137 ts_tablet_manager.cc:1909] T f352efa74d3c4da3907efdc37237211f P 201d2b6485084e7dad2cbb722af94779: Deleting consensus metadata
I20240502 20:46:08.824672 7297 tablet_replica.cc:330] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P c92e6e46782849479c8a0c6d82276fb2: stopping tablet replica
I20240502 20:46:08.824635 4507 raft_consensus.cc:2932] T c55ba788f592448d8941ff94a8cefc52 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } } }
I20240502 20:46:08.824765 4222 catalog_manager.cc:4830] TS c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827): tablet be40862863f44123916c80a1ae5b24a9 (table rebalance_test_table_12 [id=dab4c58041724066a7c74bff06307285]) successfully deleted
I20240502 20:46:08.824815 7297 raft_consensus.cc:2229] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:08.824944 7297 raft_consensus.cc:2258] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:08.825110 4219 catalog_manager.cc:4830] TS 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): tablet f352efa74d3c4da3907efdc37237211f (table rebalance_test_table_12 [id=dab4c58041724066a7c74bff06307285]) successfully deleted
I20240502 20:46:08.825285 4750 tablet_service.cc:1514] Processing DeleteTablet for tablet 6faa71d4bd2143b0a72cd86a7e1e7e12 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:08 UTC) from {username='slave'} at 127.0.0.1:38526
I20240502 20:46:08.825384 8510 tablet_replica.cc:330] T be40862863f44123916c80a1ae5b24a9 P 923cfab926e7443f8ffd7b21ad0a9d91: stopping tablet replica
I20240502 20:46:08.825461 7299 tablet_replica.cc:330] T f352efa74d3c4da3907efdc37237211f P ed47c023452a497192f35d2f617bf9bd: stopping tablet replica
I20240502 20:46:08.825516 8510 raft_consensus.cc:2229] T be40862863f44123916c80a1ae5b24a9 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:08.825580 7299 raft_consensus.cc:2229] T f352efa74d3c4da3907efdc37237211f P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:08.825649 8510 raft_consensus.cc:2258] T be40862863f44123916c80a1ae5b24a9 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:08.825739 7299 raft_consensus.cc:2258] T f352efa74d3c4da3907efdc37237211f P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:08.826239 8510 ts_tablet_manager.cc:1875] T be40862863f44123916c80a1ae5b24a9 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:08.826678 7299 ts_tablet_manager.cc:1875] T f352efa74d3c4da3907efdc37237211f P ed47c023452a497192f35d2f617bf9bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:08.827253 4487 tablet_service.cc:1514] Processing DeleteTablet for tablet 6faa71d4bd2143b0a72cd86a7e1e7e12 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:08 UTC) from {username='slave'} at 127.0.0.1:48890
I20240502 20:46:08.827474 4637 raft_consensus.cc:2932] T c55ba788f592448d8941ff94a8cefc52 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } } }
I20240502 20:46:08.828585 8510 ts_tablet_manager.cc:1888] T be40862863f44123916c80a1ae5b24a9 P 923cfab926e7443f8ffd7b21ad0a9d91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 20:46:08.828689 7299 ts_tablet_manager.cc:1888] T f352efa74d3c4da3907efdc37237211f P ed47c023452a497192f35d2f617bf9bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240502 20:46:08.828709 8510 log.cc:1196] T be40862863f44123916c80a1ae5b24a9 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal/wals/be40862863f44123916c80a1ae5b24a9
I20240502 20:46:08.828768 7299 log.cc:1196] T f352efa74d3c4da3907efdc37237211f P ed47c023452a497192f35d2f617bf9bd: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal/wals/f352efa74d3c4da3907efdc37237211f
I20240502 20:46:08.829041 8510 ts_tablet_manager.cc:1909] T be40862863f44123916c80a1ae5b24a9 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting consensus metadata
I20240502 20:46:08.829665 4228 catalog_manager.cc:5484] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2 reported cstate change: config changed from index -1 to 4, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New cstate: current_term: 1 leader_uuid: "c92e6e46782849479c8a0c6d82276fb2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:08.831960 8141 ts_tablet_manager.cc:1888] T be40862863f44123916c80a1ae5b24a9 P c3d26eafd51547acb780d4aac5ebe7cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 20:46:08.832032 8141 log.cc:1196] T be40862863f44123916c80a1ae5b24a9 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal/wals/be40862863f44123916c80a1ae5b24a9
I20240502 20:46:08.832326 8141 ts_tablet_manager.cc:1909] T be40862863f44123916c80a1ae5b24a9 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting consensus metadata
I20240502 20:46:08.832468 4219 catalog_manager.cc:4830] TS 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141): tablet be40862863f44123916c80a1ae5b24a9 (table rebalance_test_table_12 [id=dab4c58041724066a7c74bff06307285]) successfully deleted
I20240502 20:46:08.832653 8510 tablet_replica.cc:330] T f352efa74d3c4da3907efdc37237211f P 923cfab926e7443f8ffd7b21ad0a9d91: stopping tablet replica
I20240502 20:46:08.832757 8510 raft_consensus.cc:2229] T f352efa74d3c4da3907efdc37237211f P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:08.832868 8141 tablet_replica.cc:330] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P c3d26eafd51547acb780d4aac5ebe7cc: stopping tablet replica
I20240502 20:46:08.832914 8510 raft_consensus.cc:2258] T f352efa74d3c4da3907efdc37237211f P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:08.833050 4221 catalog_manager.cc:4830] TS c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097): tablet be40862863f44123916c80a1ae5b24a9 (table rebalance_test_table_12 [id=dab4c58041724066a7c74bff06307285]) successfully deleted
I20240502 20:46:08.833616 8510 ts_tablet_manager.cc:1875] T f352efa74d3c4da3907efdc37237211f P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:08.833706 7299 ts_tablet_manager.cc:1909] T f352efa74d3c4da3907efdc37237211f P ed47c023452a497192f35d2f617bf9bd: Deleting consensus metadata
I20240502 20:46:08.834337 7299 tablet_replica.cc:330] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P ed47c023452a497192f35d2f617bf9bd: stopping tablet replica
I20240502 20:46:08.834611 8141 raft_consensus.cc:2229] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:08.834991 4222 catalog_manager.cc:4830] TS ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193): tablet f352efa74d3c4da3907efdc37237211f (table rebalance_test_table_12 [id=dab4c58041724066a7c74bff06307285]) successfully deleted
I20240502 20:46:08.834532 7299 raft_consensus.cc:2229] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:08.835458 7299 raft_consensus.cc:2258] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:08.835665 8141 raft_consensus.cc:2258] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:08.836230 7297 ts_tablet_manager.cc:1875] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P c92e6e46782849479c8a0c6d82276fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:08.836663 8510 ts_tablet_manager.cc:1888] T f352efa74d3c4da3907efdc37237211f P 923cfab926e7443f8ffd7b21ad0a9d91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240502 20:46:08.836737 8510 log.cc:1196] T f352efa74d3c4da3907efdc37237211f P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal/wals/f352efa74d3c4da3907efdc37237211f
I20240502 20:46:08.837085 8510 ts_tablet_manager.cc:1909] T f352efa74d3c4da3907efdc37237211f P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting consensus metadata
I20240502 20:46:08.837917 4219 catalog_manager.cc:4830] TS 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141): tablet f352efa74d3c4da3907efdc37237211f (table rebalance_test_table_12 [id=dab4c58041724066a7c74bff06307285]) successfully deleted
I20240502 20:46:08.838402 7297 ts_tablet_manager.cc:1888] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P c92e6e46782849479c8a0c6d82276fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:08.838467 7297 log.cc:1196] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P c92e6e46782849479c8a0c6d82276fb2: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal/wals/6faa71d4bd2143b0a72cd86a7e1e7e12
I20240502 20:46:08.838728 7297 ts_tablet_manager.cc:1909] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P c92e6e46782849479c8a0c6d82276fb2: Deleting consensus metadata
I20240502 20:46:08.838991 4233 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 20:46:08.839114 4233 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 20:46:08.839476 4222 catalog_manager.cc:4830] TS c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827): tablet 6faa71d4bd2143b0a72cd86a7e1e7e12 (table rebalance_test_table_12 [id=dab4c58041724066a7c74bff06307285]) successfully deleted
I20240502 20:46:08.839753 8141 ts_tablet_manager.cc:1875] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:08.840952 4233 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: Sending DeleteTablet for 3 replicas of tablet 0dc7d9f64ba2438fb37fbe0b33f4a8be
I20240502 20:46:08.841302 7299 ts_tablet_manager.cc:1875] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P ed47c023452a497192f35d2f617bf9bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:08.841992 6809 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dc7d9f64ba2438fb37fbe0b33f4a8be with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:08 UTC) from {username='slave'} at 127.0.0.1:35290
I20240502 20:46:08.842314 8517 tablet_replica.cc:330] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P a4bf1df7fa6b4948be502c182bfbb833: stopping tablet replica
I20240502 20:46:08.842458 8141 ts_tablet_manager.cc:1888] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P c3d26eafd51547acb780d4aac5ebe7cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:08.842454 6946 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dc7d9f64ba2438fb37fbe0b33f4a8be with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:08 UTC) from {username='slave'} at 127.0.0.1:45156
I20240502 20:46:08.842649 8517 raft_consensus.cc:2229] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:08.842660 8141 log.cc:1196] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal/wals/6faa71d4bd2143b0a72cd86a7e1e7e12
I20240502 20:46:08.842783 8517 raft_consensus.cc:2258] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:08.843284 8517 ts_tablet_manager.cc:1875] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P a4bf1df7fa6b4948be502c182bfbb833: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:08.843492 8511 tablet_replica.cc:330] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P a47e5c56f13843018ac584914a8d4707: stopping tablet replica
I20240502 20:46:08.843626 8511 raft_consensus.cc:2229] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:08.843757 8511 raft_consensus.cc:2258] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:08.842583 4750 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dc7d9f64ba2438fb37fbe0b33f4a8be with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:08 UTC) from {username='slave'} at 127.0.0.1:38526
I20240502 20:46:08.844295 8511 ts_tablet_manager.cc:1875] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P a47e5c56f13843018ac584914a8d4707: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:08.846530 8517 ts_tablet_manager.cc:1888] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P a4bf1df7fa6b4948be502c182bfbb833: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:08.846599 8517 log.cc:1196] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P a4bf1df7fa6b4948be502c182bfbb833: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/wal/wals/0dc7d9f64ba2438fb37fbe0b33f4a8be
I20240502 20:46:08.846845 8517 ts_tablet_manager.cc:1909] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P a4bf1df7fa6b4948be502c182bfbb833: Deleting consensus metadata
I20240502 20:46:08.847797 8518 ts_tablet_manager.cc:902] T d6582234196c4fe4a82b3182787700de P 7845c397230d49358ba2fad9db53f97c: Initiating tablet copy from peer c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097)
I20240502 20:46:08.847872 4220 catalog_manager.cc:4830] TS a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867): tablet 0dc7d9f64ba2438fb37fbe0b33f4a8be (table rebalancer_dynamic_table_new_name [id=4c3872020dc144bd8931994dced22724]) successfully deleted
I20240502 20:46:08.848418 8518 tablet_copy_client.cc:308] T d6582234196c4fe4a82b3182787700de P 7845c397230d49358ba2fad9db53f97c: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.66:42097
I20240502 20:46:08.848856 4528 tablet_copy_service.cc:140] P c3d26eafd51547acb780d4aac5ebe7cc: Received BeginTabletCopySession request for tablet d6582234196c4fe4a82b3182787700de from peer 7845c397230d49358ba2fad9db53f97c ({username='slave'} at 127.30.32.69:48755)
I20240502 20:46:08.848951 4528 tablet_copy_service.cc:161] P c3d26eafd51547acb780d4aac5ebe7cc: Beginning new tablet copy session on tablet d6582234196c4fe4a82b3182787700de from peer 7845c397230d49358ba2fad9db53f97c at {username='slave'} at 127.30.32.69:48755: session id = 7845c397230d49358ba2fad9db53f97c-d6582234196c4fe4a82b3182787700de
I20240502 20:46:08.849745 8453 ts_tablet_manager.cc:902] T 0cf7e95cb91b4e74b14365fcc6b7f968 P a47e5c56f13843018ac584914a8d4707: Initiating tablet copy from peer 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:46:08.849865 4528 tablet_copy_source_session.cc:205] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:08.850174 8453 tablet_copy_client.cc:308] T 0cf7e95cb91b4e74b14365fcc6b7f968 P a47e5c56f13843018ac584914a8d4707: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.67:42141
I20240502 20:46:08.850620 4658 tablet_copy_service.cc:140] P 923cfab926e7443f8ffd7b21ad0a9d91: Received BeginTabletCopySession request for tablet 0cf7e95cb91b4e74b14365fcc6b7f968 from peer a47e5c56f13843018ac584914a8d4707 ({username='slave'} at 127.30.32.71:59213)
I20240502 20:46:08.850741 4658 tablet_copy_service.cc:161] P 923cfab926e7443f8ffd7b21ad0a9d91: Beginning new tablet copy session on tablet 0cf7e95cb91b4e74b14365fcc6b7f968 from peer a47e5c56f13843018ac584914a8d4707 at {username='slave'} at 127.30.32.71:59213: session id = a47e5c56f13843018ac584914a8d4707-0cf7e95cb91b4e74b14365fcc6b7f968
I20240502 20:46:08.850859 8518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6582234196c4fe4a82b3182787700de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:08.851649 4658 tablet_copy_source_session.cc:205] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:08.850312 8511 ts_tablet_manager.cc:1888] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P a47e5c56f13843018ac584914a8d4707: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:08.852243 8511 log.cc:1196] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P a47e5c56f13843018ac584914a8d4707: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/wal/wals/0dc7d9f64ba2438fb37fbe0b33f4a8be
I20240502 20:46:08.852505 8511 ts_tablet_manager.cc:1909] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P a47e5c56f13843018ac584914a8d4707: Deleting consensus metadata
I20240502 20:46:08.852507 8453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cf7e95cb91b4e74b14365fcc6b7f968. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:08.852871 8518 tablet_copy_client.cc:783] T d6582234196c4fe4a82b3182787700de P 7845c397230d49358ba2fad9db53f97c: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:08.854229 7299 ts_tablet_manager.cc:1888] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P ed47c023452a497192f35d2f617bf9bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:08.854291 7299 log.cc:1196] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P ed47c023452a497192f35d2f617bf9bd: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal/wals/6faa71d4bd2143b0a72cd86a7e1e7e12
I20240502 20:46:08.854525 7299 ts_tablet_manager.cc:1909] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P ed47c023452a497192f35d2f617bf9bd: Deleting consensus metadata
I20240502 20:46:08.854604 8453 tablet_copy_client.cc:783] T 0cf7e95cb91b4e74b14365fcc6b7f968 P a47e5c56f13843018ac584914a8d4707: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:08.854725 8453 tablet_copy_client.cc:647] T 0cf7e95cb91b4e74b14365fcc6b7f968 P a47e5c56f13843018ac584914a8d4707: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:08.855018 4222 catalog_manager.cc:4830] TS a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): tablet 0dc7d9f64ba2438fb37fbe0b33f4a8be (table rebalancer_dynamic_table_new_name [id=4c3872020dc144bd8931994dced22724]) successfully deleted
I20240502 20:46:08.855297 8141 ts_tablet_manager.cc:1909] T 6faa71d4bd2143b0a72cd86a7e1e7e12 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting consensus metadata
I20240502 20:46:08.855515 4222 catalog_manager.cc:4830] TS ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193): tablet 6faa71d4bd2143b0a72cd86a7e1e7e12 (table rebalance_test_table_12 [id=dab4c58041724066a7c74bff06307285]) successfully deleted
I20240502 20:46:08.855566 7299 tablet_replica.cc:330] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P ed47c023452a497192f35d2f617bf9bd: stopping tablet replica
I20240502 20:46:08.855710 7299 raft_consensus.cc:2229] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:08.855986 4221 catalog_manager.cc:4830] TS c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097): tablet 6faa71d4bd2143b0a72cd86a7e1e7e12 (table rebalance_test_table_12 [id=dab4c58041724066a7c74bff06307285]) successfully deleted
I20240502 20:46:08.855918 7299 raft_consensus.cc:2258] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:08.856796 7299 ts_tablet_manager.cc:1875] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P ed47c023452a497192f35d2f617bf9bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:08.858464 6967 consensus_queue.cc:237] T 747e7331d7ad4dd4b132672254037e1b P a47e5c56f13843018ac584914a8d4707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } }
I20240502 20:46:08.858877 7299 ts_tablet_manager.cc:1888] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P ed47c023452a497192f35d2f617bf9bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:08.858934 7299 log.cc:1196] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P ed47c023452a497192f35d2f617bf9bd: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal/wals/0dc7d9f64ba2438fb37fbe0b33f4a8be
I20240502 20:46:08.859158 7299 ts_tablet_manager.cc:1909] T 0dc7d9f64ba2438fb37fbe0b33f4a8be P ed47c023452a497192f35d2f617bf9bd: Deleting consensus metadata
I20240502 20:46:08.859722 4222 catalog_manager.cc:4830] TS ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193): tablet 0dc7d9f64ba2438fb37fbe0b33f4a8be (table rebalancer_dynamic_table_new_name [id=4c3872020dc144bd8931994dced22724]) successfully deleted
I20240502 20:46:08.859972 6831 raft_consensus.cc:1261] T 747e7331d7ad4dd4b132672254037e1b P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Refusing update from remote peer a47e5c56f13843018ac584914a8d4707: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:08.860021 6702 raft_consensus.cc:1261] T 747e7331d7ad4dd4b132672254037e1b P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Refusing update from remote peer a47e5c56f13843018ac584914a8d4707: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:08.860421 7310 consensus_queue.cc:1035] T 747e7331d7ad4dd4b132672254037e1b P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:46:08.860653 8304 consensus_queue.cc:1035] T 747e7331d7ad4dd4b132672254037e1b P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 20:46:08.861660 6902 consensus_peers.cc:485] T 747e7331d7ad4dd4b132672254037e1b P a47e5c56f13843018ac584914a8d4707 -> Peer c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097): Couldn't send request to peer c3d26eafd51547acb780d4aac5ebe7cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 747e7331d7ad4dd4b132672254037e1b. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:08.861788 8453 tablet_copy_client.cc:523] T 0cf7e95cb91b4e74b14365fcc6b7f968 P a47e5c56f13843018ac584914a8d4707: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:08.863257 8453 tablet_bootstrap.cc:492] T 0cf7e95cb91b4e74b14365fcc6b7f968 P a47e5c56f13843018ac584914a8d4707: Bootstrap starting.
I20240502 20:46:08.863376 8304 raft_consensus.cc:2932] T 747e7331d7ad4dd4b132672254037e1b P a47e5c56f13843018ac584914a8d4707 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } } }
I20240502 20:46:08.864051 8523 ts_tablet_manager.cc:902] T 180902058815460189f75de4704d1e84 P 7845c397230d49358ba2fad9db53f97c: Initiating tablet copy from peer a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867)
I20240502 20:46:08.864229 8518 tablet_copy_client.cc:647] T d6582234196c4fe4a82b3182787700de P 7845c397230d49358ba2fad9db53f97c: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:08.864604 8523 tablet_copy_client.cc:308] T 180902058815460189f75de4704d1e84 P 7845c397230d49358ba2fad9db53f97c: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.70:41867
I20240502 20:46:08.867316 4233 catalog_manager.cc:5484] T 747e7331d7ad4dd4b132672254037e1b P a47e5c56f13843018ac584914a8d4707 reported cstate change: config changed from index -1 to 2, NON_VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) added. New cstate: current_term: 1 leader_uuid: "a47e5c56f13843018ac584914a8d4707" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:08.867336 6831 raft_consensus.cc:2932] T 747e7331d7ad4dd4b132672254037e1b P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } } }
I20240502 20:46:08.868613 6702 raft_consensus.cc:2932] T 747e7331d7ad4dd4b132672254037e1b P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } } }
I20240502 20:46:08.874073 8518 tablet_copy_client.cc:523] T d6582234196c4fe4a82b3182787700de P 7845c397230d49358ba2fad9db53f97c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:08.875600 8518 tablet_bootstrap.cc:492] T d6582234196c4fe4a82b3182787700de P 7845c397230d49358ba2fad9db53f97c: Bootstrap starting.
I20240502 20:46:08.879638 6848 tablet_copy_service.cc:140] P a4bf1df7fa6b4948be502c182bfbb833: Received BeginTabletCopySession request for tablet 180902058815460189f75de4704d1e84 from peer 7845c397230d49358ba2fad9db53f97c ({username='slave'} at 127.30.32.69:38627)
I20240502 20:46:08.879781 6848 tablet_copy_service.cc:161] P a4bf1df7fa6b4948be502c182bfbb833: Beginning new tablet copy session on tablet 180902058815460189f75de4704d1e84 from peer 7845c397230d49358ba2fad9db53f97c at {username='slave'} at 127.30.32.69:38627: session id = 7845c397230d49358ba2fad9db53f97c-180902058815460189f75de4704d1e84
I20240502 20:46:08.881160 6848 tablet_copy_source_session.cc:205] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:08.882346 8523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 180902058815460189f75de4704d1e84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:08.903564 8523 tablet_copy_client.cc:783] T 180902058815460189f75de4704d1e84 P 7845c397230d49358ba2fad9db53f97c: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:08.903759 8523 tablet_copy_client.cc:647] T 180902058815460189f75de4704d1e84 P 7845c397230d49358ba2fad9db53f97c: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:08.908674 8453 tablet_bootstrap.cc:492] T 0cf7e95cb91b4e74b14365fcc6b7f968 P a47e5c56f13843018ac584914a8d4707: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:08.909091 8453 tablet_bootstrap.cc:492] T 0cf7e95cb91b4e74b14365fcc6b7f968 P a47e5c56f13843018ac584914a8d4707: Bootstrap complete.
I20240502 20:46:08.909372 8453 ts_tablet_manager.cc:1367] T 0cf7e95cb91b4e74b14365fcc6b7f968 P a47e5c56f13843018ac584914a8d4707: Time spent bootstrapping tablet: real 0.046s user 0.020s sys 0.007s
I20240502 20:46:08.909657 8523 tablet_copy_client.cc:523] T 180902058815460189f75de4704d1e84 P 7845c397230d49358ba2fad9db53f97c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:08.910744 8453 raft_consensus.cc:348] T 0cf7e95cb91b4e74b14365fcc6b7f968 P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } }
I20240502 20:46:08.910871 8453 raft_consensus.cc:729] T 0cf7e95cb91b4e74b14365fcc6b7f968 P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Initialized, Role: LEARNER
I20240502 20:46:08.911005 8453 consensus_queue.cc:260] T 0cf7e95cb91b4e74b14365fcc6b7f968 P a47e5c56f13843018ac584914a8d4707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } }
I20240502 20:46:08.912973 8453 ts_tablet_manager.cc:1398] T 0cf7e95cb91b4e74b14365fcc6b7f968 P a47e5c56f13843018ac584914a8d4707: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 20:46:08.913313 8556 ts_tablet_manager.cc:902] T ed6386b6777c4d50b29ff8cd49894eba P a4bf1df7fa6b4948be502c182bfbb833: Initiating tablet copy from peer 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:46:08.913501 4658 tablet_copy_service.cc:342] P 923cfab926e7443f8ffd7b21ad0a9d91: Request end of tablet copy session a47e5c56f13843018ac584914a8d4707-0cf7e95cb91b4e74b14365fcc6b7f968 received from {username='slave'} at 127.30.32.71:59213
I20240502 20:46:08.913589 4658 tablet_copy_service.cc:433] P 923cfab926e7443f8ffd7b21ad0a9d91: ending tablet copy session a47e5c56f13843018ac584914a8d4707-0cf7e95cb91b4e74b14365fcc6b7f968 on tablet 0cf7e95cb91b4e74b14365fcc6b7f968 with peer a47e5c56f13843018ac584914a8d4707
I20240502 20:46:08.914719 8556 tablet_copy_client.cc:308] T ed6386b6777c4d50b29ff8cd49894eba P a4bf1df7fa6b4948be502c182bfbb833: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.67:42141
I20240502 20:46:08.915146 4658 tablet_copy_service.cc:140] P 923cfab926e7443f8ffd7b21ad0a9d91: Received BeginTabletCopySession request for tablet ed6386b6777c4d50b29ff8cd49894eba from peer a4bf1df7fa6b4948be502c182bfbb833 ({username='slave'} at 127.30.32.70:32901)
I20240502 20:46:08.915243 4658 tablet_copy_service.cc:161] P 923cfab926e7443f8ffd7b21ad0a9d91: Beginning new tablet copy session on tablet ed6386b6777c4d50b29ff8cd49894eba from peer a4bf1df7fa6b4948be502c182bfbb833 at {username='slave'} at 127.30.32.70:32901: session id = a4bf1df7fa6b4948be502c182bfbb833-ed6386b6777c4d50b29ff8cd49894eba
I20240502 20:46:08.916057 4658 tablet_copy_source_session.cc:205] T ed6386b6777c4d50b29ff8cd49894eba P 923cfab926e7443f8ffd7b21ad0a9d91: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:08.916888 8556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed6386b6777c4d50b29ff8cd49894eba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:08.917768 8523 tablet_bootstrap.cc:492] T 180902058815460189f75de4704d1e84 P 7845c397230d49358ba2fad9db53f97c: Bootstrap starting.
I20240502 20:46:08.919314 8556 tablet_copy_client.cc:783] T ed6386b6777c4d50b29ff8cd49894eba P a4bf1df7fa6b4948be502c182bfbb833: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:08.919540 8556 tablet_copy_client.cc:647] T ed6386b6777c4d50b29ff8cd49894eba P a4bf1df7fa6b4948be502c182bfbb833: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:08.920639 8556 tablet_copy_client.cc:523] T ed6386b6777c4d50b29ff8cd49894eba P a4bf1df7fa6b4948be502c182bfbb833: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:08.921998 8556 tablet_bootstrap.cc:492] T ed6386b6777c4d50b29ff8cd49894eba P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap starting.
I20240502 20:46:08.928301 8523 tablet_bootstrap.cc:492] T 180902058815460189f75de4704d1e84 P 7845c397230d49358ba2fad9db53f97c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:08.928594 8523 tablet_bootstrap.cc:492] T 180902058815460189f75de4704d1e84 P 7845c397230d49358ba2fad9db53f97c: Bootstrap complete.
I20240502 20:46:08.928791 8523 ts_tablet_manager.cc:1367] T 180902058815460189f75de4704d1e84 P 7845c397230d49358ba2fad9db53f97c: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.001s
I20240502 20:46:08.929458 8523 raft_consensus.cc:348] T 180902058815460189f75de4704d1e84 P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } }
I20240502 20:46:08.929618 8523 raft_consensus.cc:729] T 180902058815460189f75de4704d1e84 P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Initialized, Role: LEARNER
I20240502 20:46:08.929733 8523 consensus_queue.cc:260] T 180902058815460189f75de4704d1e84 P 7845c397230d49358ba2fad9db53f97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } }
I20240502 20:46:08.930032 8523 ts_tablet_manager.cc:1398] T 180902058815460189f75de4704d1e84 P 7845c397230d49358ba2fad9db53f97c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:08.931401 6848 tablet_copy_service.cc:342] P a4bf1df7fa6b4948be502c182bfbb833: Request end of tablet copy session 7845c397230d49358ba2fad9db53f97c-180902058815460189f75de4704d1e84 received from {username='slave'} at 127.30.32.69:38627
I20240502 20:46:08.931504 6848 tablet_copy_service.cc:433] P a4bf1df7fa6b4948be502c182bfbb833: ending tablet copy session 7845c397230d49358ba2fad9db53f97c-180902058815460189f75de4704d1e84 on tablet 180902058815460189f75de4704d1e84 with peer 7845c397230d49358ba2fad9db53f97c
I20240502 20:46:08.936949 8518 tablet_bootstrap.cc:492] T d6582234196c4fe4a82b3182787700de P 7845c397230d49358ba2fad9db53f97c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:08.937233 8518 tablet_bootstrap.cc:492] T d6582234196c4fe4a82b3182787700de P 7845c397230d49358ba2fad9db53f97c: Bootstrap complete.
I20240502 20:46:08.937405 8518 ts_tablet_manager.cc:1367] T d6582234196c4fe4a82b3182787700de P 7845c397230d49358ba2fad9db53f97c: Time spent bootstrapping tablet: real 0.062s user 0.026s sys 0.004s
I20240502 20:46:08.937901 8518 raft_consensus.cc:348] T d6582234196c4fe4a82b3182787700de P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } }
I20240502 20:46:08.938055 8518 raft_consensus.cc:729] T d6582234196c4fe4a82b3182787700de P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Initialized, Role: LEARNER
I20240502 20:46:08.938175 8518 consensus_queue.cc:260] T d6582234196c4fe4a82b3182787700de P 7845c397230d49358ba2fad9db53f97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } }
I20240502 20:46:08.938469 8518 ts_tablet_manager.cc:1398] T d6582234196c4fe4a82b3182787700de P 7845c397230d49358ba2fad9db53f97c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:08.939038 4528 tablet_copy_service.cc:342] P c3d26eafd51547acb780d4aac5ebe7cc: Request end of tablet copy session 7845c397230d49358ba2fad9db53f97c-d6582234196c4fe4a82b3182787700de received from {username='slave'} at 127.30.32.69:48755
I20240502 20:46:08.939124 4528 tablet_copy_service.cc:433] P c3d26eafd51547acb780d4aac5ebe7cc: ending tablet copy session 7845c397230d49358ba2fad9db53f97c-d6582234196c4fe4a82b3182787700de on tablet d6582234196c4fe4a82b3182787700de with peer 7845c397230d49358ba2fad9db53f97c
I20240502 20:46:08.942952 4233 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41812:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 20:46:08.948408 6809 tablet_service.cc:1467] Processing CreateTablet for tablet a88bdfa2bca8401d99eec723457a1125 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9a174cd46c574a628620ba4cba9d6466]), partition=
I20240502 20:46:08.948712 6809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a88bdfa2bca8401d99eec723457a1125. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:08.948938 7099 raft_consensus.cc:1261] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 201d2b6485084e7dad2cbb722af94779 [term 2 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 20:46:08.949337 4618 tablet_service.cc:1467] Processing CreateTablet for tablet a88bdfa2bca8401d99eec723457a1125 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9a174cd46c574a628620ba4cba9d6466]), partition=
I20240502 20:46:08.949577 4618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a88bdfa2bca8401d99eec723457a1125. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:08.950803 8579 tablet_bootstrap.cc:492] T a88bdfa2bca8401d99eec723457a1125 P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap starting.
I20240502 20:46:08.951454 8580 tablet_bootstrap.cc:492] T a88bdfa2bca8401d99eec723457a1125 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:46:08.951763 8579 tablet_bootstrap.cc:654] T a88bdfa2bca8401d99eec723457a1125 P a4bf1df7fa6b4948be502c182bfbb833: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:08.951776 6682 tablet_service.cc:1467] Processing CreateTablet for tablet a88bdfa2bca8401d99eec723457a1125 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9a174cd46c574a628620ba4cba9d6466]), partition=
I20240502 20:46:08.952009 6682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a88bdfa2bca8401d99eec723457a1125. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:08.952328 8580 tablet_bootstrap.cc:654] T a88bdfa2bca8401d99eec723457a1125 P 923cfab926e7443f8ffd7b21ad0a9d91: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:08.953392 8580 tablet_bootstrap.cc:492] T a88bdfa2bca8401d99eec723457a1125 P 923cfab926e7443f8ffd7b21ad0a9d91: No bootstrap required, opened a new log
I20240502 20:46:08.953470 8580 ts_tablet_manager.cc:1367] T a88bdfa2bca8401d99eec723457a1125 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:08.953861 8580 raft_consensus.cc:348] T a88bdfa2bca8401d99eec723457a1125 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:08.953963 8580 raft_consensus.cc:374] T a88bdfa2bca8401d99eec723457a1125 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:08.953994 8580 raft_consensus.cc:729] T a88bdfa2bca8401d99eec723457a1125 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: FOLLOWER
I20240502 20:46:08.954126 8580 consensus_queue.cc:260] T a88bdfa2bca8401d99eec723457a1125 P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:08.954435 8580 ts_tablet_manager.cc:1398] T a88bdfa2bca8401d99eec723457a1125 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:08.954962 8583 tablet_bootstrap.cc:492] T a88bdfa2bca8401d99eec723457a1125 P 7845c397230d49358ba2fad9db53f97c: Bootstrap starting.
I20240502 20:46:08.956598 8583 tablet_bootstrap.cc:654] T a88bdfa2bca8401d99eec723457a1125 P 7845c397230d49358ba2fad9db53f97c: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:08.957290 8579 tablet_bootstrap.cc:492] T a88bdfa2bca8401d99eec723457a1125 P a4bf1df7fa6b4948be502c182bfbb833: No bootstrap required, opened a new log
I20240502 20:46:08.957382 8579 ts_tablet_manager.cc:1367] T a88bdfa2bca8401d99eec723457a1125 P a4bf1df7fa6b4948be502c182bfbb833: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 20:46:08.957649 8583 tablet_bootstrap.cc:492] T a88bdfa2bca8401d99eec723457a1125 P 7845c397230d49358ba2fad9db53f97c: No bootstrap required, opened a new log
I20240502 20:46:08.957712 8583 ts_tablet_manager.cc:1367] T a88bdfa2bca8401d99eec723457a1125 P 7845c397230d49358ba2fad9db53f97c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:46:08.957967 8579 raft_consensus.cc:348] T a88bdfa2bca8401d99eec723457a1125 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:08.958089 8579 raft_consensus.cc:374] T a88bdfa2bca8401d99eec723457a1125 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:08.958039 8583 raft_consensus.cc:348] T a88bdfa2bca8401d99eec723457a1125 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:08.958132 8583 raft_consensus.cc:374] T a88bdfa2bca8401d99eec723457a1125 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:08.958127 8579 raft_consensus.cc:729] T a88bdfa2bca8401d99eec723457a1125 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Initialized, Role: FOLLOWER
I20240502 20:46:08.958159 8583 raft_consensus.cc:729] T a88bdfa2bca8401d99eec723457a1125 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Initialized, Role: FOLLOWER
I20240502 20:46:08.958288 8579 consensus_queue.cc:260] T a88bdfa2bca8401d99eec723457a1125 P a4bf1df7fa6b4948be502c182bfbb833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:08.958283 8583 consensus_queue.cc:260] T a88bdfa2bca8401d99eec723457a1125 P 7845c397230d49358ba2fad9db53f97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:08.958567 8579 ts_tablet_manager.cc:1398] T a88bdfa2bca8401d99eec723457a1125 P a4bf1df7fa6b4948be502c182bfbb833: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:08.958581 8583 ts_tablet_manager.cc:1398] T a88bdfa2bca8401d99eec723457a1125 P 7845c397230d49358ba2fad9db53f97c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:08.958984 7937 consensus_queue.cc:1035] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:08.976389 8556 tablet_bootstrap.cc:492] T ed6386b6777c4d50b29ff8cd49894eba P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:08.976728 8556 tablet_bootstrap.cc:492] T ed6386b6777c4d50b29ff8cd49894eba P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap complete.
I20240502 20:46:08.976897 8556 ts_tablet_manager.cc:1367] T ed6386b6777c4d50b29ff8cd49894eba P a4bf1df7fa6b4948be502c182bfbb833: Time spent bootstrapping tablet: real 0.055s user 0.021s sys 0.000s
I20240502 20:46:08.977391 8556 raft_consensus.cc:348] T ed6386b6777c4d50b29ff8cd49894eba P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } }
I20240502 20:46:08.977555 8556 raft_consensus.cc:729] T ed6386b6777c4d50b29ff8cd49894eba P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Initialized, Role: LEARNER
I20240502 20:46:08.977694 8556 consensus_queue.cc:260] T ed6386b6777c4d50b29ff8cd49894eba P a4bf1df7fa6b4948be502c182bfbb833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } }
I20240502 20:46:08.978027 8556 ts_tablet_manager.cc:1398] T ed6386b6777c4d50b29ff8cd49894eba P a4bf1df7fa6b4948be502c182bfbb833: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240502 20:46:08.978649 4658 tablet_copy_service.cc:342] P 923cfab926e7443f8ffd7b21ad0a9d91: Request end of tablet copy session a4bf1df7fa6b4948be502c182bfbb833-ed6386b6777c4d50b29ff8cd49894eba received from {username='slave'} at 127.30.32.70:32901
I20240502 20:46:08.978739 4658 tablet_copy_service.cc:433] P 923cfab926e7443f8ffd7b21ad0a9d91: ending tablet copy session a4bf1df7fa6b4948be502c182bfbb833-ed6386b6777c4d50b29ff8cd49894eba on tablet ed6386b6777c4d50b29ff8cd49894eba with peer a4bf1df7fa6b4948be502c182bfbb833
I20240502 20:46:08.988538 8528 consensus_queue.cc:1035] T 8602022a45e1482ebf8dab9930ce8209 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:08.992587 8304 consensus_queue.cc:1035] T 8602022a45e1482ebf8dab9930ce8209 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:09.005540 8518 ts_tablet_manager.cc:902] T 04c72d7d63574dea99cb2f7c14199a73 P 7845c397230d49358ba2fad9db53f97c: Initiating tablet copy from peer c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:46:09.006115 8518 tablet_copy_client.cc:308] T 04c72d7d63574dea99cb2f7c14199a73 P 7845c397230d49358ba2fad9db53f97c: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.65:33827
I20240502 20:46:09.009974 4397 tablet_copy_service.cc:140] P c92e6e46782849479c8a0c6d82276fb2: Received BeginTabletCopySession request for tablet 04c72d7d63574dea99cb2f7c14199a73 from peer 7845c397230d49358ba2fad9db53f97c ({username='slave'} at 127.30.32.69:38945)
I20240502 20:46:09.010089 4397 tablet_copy_service.cc:161] P c92e6e46782849479c8a0c6d82276fb2: Beginning new tablet copy session on tablet 04c72d7d63574dea99cb2f7c14199a73 from peer 7845c397230d49358ba2fad9db53f97c at {username='slave'} at 127.30.32.69:38945: session id = 7845c397230d49358ba2fad9db53f97c-04c72d7d63574dea99cb2f7c14199a73
I20240502 20:46:09.011085 4397 tablet_copy_source_session.cc:205] T 04c72d7d63574dea99cb2f7c14199a73 P c92e6e46782849479c8a0c6d82276fb2: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:09.012012 8518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04c72d7d63574dea99cb2f7c14199a73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:09.014425 8518 tablet_copy_client.cc:783] T 04c72d7d63574dea99cb2f7c14199a73 P 7845c397230d49358ba2fad9db53f97c: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:09.014591 8518 tablet_copy_client.cc:647] T 04c72d7d63574dea99cb2f7c14199a73 P 7845c397230d49358ba2fad9db53f97c: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:09.015791 8518 tablet_copy_client.cc:523] T 04c72d7d63574dea99cb2f7c14199a73 P 7845c397230d49358ba2fad9db53f97c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:09.017115 8518 tablet_bootstrap.cc:492] T 04c72d7d63574dea99cb2f7c14199a73 P 7845c397230d49358ba2fad9db53f97c: Bootstrap starting.
I20240502 20:46:09.041290 4637 consensus_queue.cc:237] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } }
I20240502 20:46:09.043017 6831 raft_consensus.cc:1261] T 7abfd61f55554042978b50761cbd6ab5 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:09.043594 8523 ts_tablet_manager.cc:902] T a55d5b8b9ed04ca39888b89b82a3f833 P 7845c397230d49358ba2fad9db53f97c: Initiating tablet copy from peer ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:46:09.044080 8523 tablet_copy_client.cc:308] T a55d5b8b9ed04ca39888b89b82a3f833 P 7845c397230d49358ba2fad9db53f97c: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.68:38193
I20240502 20:46:09.044148 8237 consensus_queue.cc:1035] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:09.044458 4790 tablet_copy_service.cc:140] P ed47c023452a497192f35d2f617bf9bd: Received BeginTabletCopySession request for tablet a55d5b8b9ed04ca39888b89b82a3f833 from peer 7845c397230d49358ba2fad9db53f97c ({username='slave'} at 127.30.32.69:43573)
I20240502 20:46:09.044569 4790 tablet_copy_service.cc:161] P ed47c023452a497192f35d2f617bf9bd: Beginning new tablet copy session on tablet a55d5b8b9ed04ca39888b89b82a3f833 from peer 7845c397230d49358ba2fad9db53f97c at {username='slave'} at 127.30.32.69:43573: session id = 7845c397230d49358ba2fad9db53f97c-a55d5b8b9ed04ca39888b89b82a3f833
I20240502 20:46:09.044665 4768 raft_consensus.cc:1261] T 7abfd61f55554042978b50761cbd6ab5 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 20:46:09.045137 4575 consensus_peers.cc:485] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7abfd61f55554042978b50761cbd6ab5. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:09.045248 6613 consensus_queue.cc:1035] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:09.045495 4790 tablet_copy_source_session.cc:205] T a55d5b8b9ed04ca39888b89b82a3f833 P ed47c023452a497192f35d2f617bf9bd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:09.046358 8523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a55d5b8b9ed04ca39888b89b82a3f833. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:09.047014 8518 tablet_bootstrap.cc:492] T 04c72d7d63574dea99cb2f7c14199a73 P 7845c397230d49358ba2fad9db53f97c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:09.047441 8518 tablet_bootstrap.cc:492] T 04c72d7d63574dea99cb2f7c14199a73 P 7845c397230d49358ba2fad9db53f97c: Bootstrap complete.
I20240502 20:46:09.047624 8518 ts_tablet_manager.cc:1367] T 04c72d7d63574dea99cb2f7c14199a73 P 7845c397230d49358ba2fad9db53f97c: Time spent bootstrapping tablet: real 0.031s user 0.022s sys 0.003s
I20240502 20:46:09.047582 8237 raft_consensus.cc:2932] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } } }
I20240502 20:46:09.048066 8518 raft_consensus.cc:348] T 04c72d7d63574dea99cb2f7c14199a73 P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } }
I20240502 20:46:09.048185 8518 raft_consensus.cc:729] T 04c72d7d63574dea99cb2f7c14199a73 P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Initialized, Role: LEARNER
I20240502 20:46:09.048301 8518 consensus_queue.cc:260] T 04c72d7d63574dea99cb2f7c14199a73 P 7845c397230d49358ba2fad9db53f97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } }
I20240502 20:46:09.048574 8523 tablet_copy_client.cc:783] T a55d5b8b9ed04ca39888b89b82a3f833 P 7845c397230d49358ba2fad9db53f97c: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:09.048712 8523 tablet_copy_client.cc:647] T a55d5b8b9ed04ca39888b89b82a3f833 P 7845c397230d49358ba2fad9db53f97c: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:09.049213 8518 ts_tablet_manager.cc:1398] T 04c72d7d63574dea99cb2f7c14199a73 P 7845c397230d49358ba2fad9db53f97c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:09.049750 4397 tablet_copy_service.cc:342] P c92e6e46782849479c8a0c6d82276fb2: Request end of tablet copy session 7845c397230d49358ba2fad9db53f97c-04c72d7d63574dea99cb2f7c14199a73 received from {username='slave'} at 127.30.32.69:38945
I20240502 20:46:09.049846 4397 tablet_copy_service.cc:433] P c92e6e46782849479c8a0c6d82276fb2: ending tablet copy session 7845c397230d49358ba2fad9db53f97c-04c72d7d63574dea99cb2f7c14199a73 on tablet 04c72d7d63574dea99cb2f7c14199a73 with peer 7845c397230d49358ba2fad9db53f97c
I20240502 20:46:09.049868 6831 raft_consensus.cc:2932] T 7abfd61f55554042978b50761cbd6ab5 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } } }
I20240502 20:46:09.050871 4233 catalog_manager.cc:5484] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index 6 to 7, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:09.051347 4768 raft_consensus.cc:2932] T 7abfd61f55554042978b50761cbd6ab5 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } } }
I20240502 20:46:09.057461 8556 ts_tablet_manager.cc:902] T 5053a7afb4994ecdbc92b129c58903cf P a4bf1df7fa6b4948be502c182bfbb833: Initiating tablet copy from peer c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097)
I20240502 20:46:09.057945 8556 tablet_copy_client.cc:308] T 5053a7afb4994ecdbc92b129c58903cf P a4bf1df7fa6b4948be502c182bfbb833: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.66:42097
I20240502 20:46:09.058630 4528 tablet_copy_service.cc:140] P c3d26eafd51547acb780d4aac5ebe7cc: Received BeginTabletCopySession request for tablet 5053a7afb4994ecdbc92b129c58903cf from peer a4bf1df7fa6b4948be502c182bfbb833 ({username='slave'} at 127.30.32.70:46909)
I20240502 20:46:09.058729 4528 tablet_copy_service.cc:161] P c3d26eafd51547acb780d4aac5ebe7cc: Beginning new tablet copy session on tablet 5053a7afb4994ecdbc92b129c58903cf from peer a4bf1df7fa6b4948be502c182bfbb833 at {username='slave'} at 127.30.32.70:46909: session id = a4bf1df7fa6b4948be502c182bfbb833-5053a7afb4994ecdbc92b129c58903cf
I20240502 20:46:09.059718 4528 tablet_copy_source_session.cc:205] T 5053a7afb4994ecdbc92b129c58903cf P c3d26eafd51547acb780d4aac5ebe7cc: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:09.060725 8556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5053a7afb4994ecdbc92b129c58903cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:09.062666 8523 tablet_copy_client.cc:523] T a55d5b8b9ed04ca39888b89b82a3f833 P 7845c397230d49358ba2fad9db53f97c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:09.063935 8523 tablet_bootstrap.cc:492] T a55d5b8b9ed04ca39888b89b82a3f833 P 7845c397230d49358ba2fad9db53f97c: Bootstrap starting.
I20240502 20:46:09.064078 8556 tablet_copy_client.cc:783] T 5053a7afb4994ecdbc92b129c58903cf P a4bf1df7fa6b4948be502c182bfbb833: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:09.064244 8556 tablet_copy_client.cc:647] T 5053a7afb4994ecdbc92b129c58903cf P a4bf1df7fa6b4948be502c182bfbb833: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:09.065522 8556 tablet_copy_client.cc:523] T 5053a7afb4994ecdbc92b129c58903cf P a4bf1df7fa6b4948be502c182bfbb833: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:09.066783 8556 tablet_bootstrap.cc:492] T 5053a7afb4994ecdbc92b129c58903cf P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap starting.
I20240502 20:46:09.078413 4507 raft_consensus.cc:1261] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc [term 2 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 20:46:09.078930 7937 consensus_queue.cc:1035] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:09.088690 8523 tablet_bootstrap.cc:492] T a55d5b8b9ed04ca39888b89b82a3f833 P 7845c397230d49358ba2fad9db53f97c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:09.089056 8523 tablet_bootstrap.cc:492] T a55d5b8b9ed04ca39888b89b82a3f833 P 7845c397230d49358ba2fad9db53f97c: Bootstrap complete.
I20240502 20:46:09.089407 8523 ts_tablet_manager.cc:1367] T a55d5b8b9ed04ca39888b89b82a3f833 P 7845c397230d49358ba2fad9db53f97c: Time spent bootstrapping tablet: real 0.026s user 0.026s sys 0.000s
I20240502 20:46:09.089896 8523 raft_consensus.cc:348] T a55d5b8b9ed04ca39888b89b82a3f833 P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } }
I20240502 20:46:09.090142 4638 raft_consensus.cc:1261] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 2 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 20:46:09.090029 8523 raft_consensus.cc:729] T a55d5b8b9ed04ca39888b89b82a3f833 P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Initialized, Role: LEARNER
I20240502 20:46:09.090294 8523 consensus_queue.cc:260] T a55d5b8b9ed04ca39888b89b82a3f833 P 7845c397230d49358ba2fad9db53f97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } }
I20240502 20:46:09.090608 8523 ts_tablet_manager.cc:1398] T a55d5b8b9ed04ca39888b89b82a3f833 P 7845c397230d49358ba2fad9db53f97c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:09.091331 4790 tablet_copy_service.cc:342] P ed47c023452a497192f35d2f617bf9bd: Request end of tablet copy session 7845c397230d49358ba2fad9db53f97c-a55d5b8b9ed04ca39888b89b82a3f833 received from {username='slave'} at 127.30.32.69:43573
I20240502 20:46:09.091424 4790 tablet_copy_service.cc:433] P ed47c023452a497192f35d2f617bf9bd: ending tablet copy session 7845c397230d49358ba2fad9db53f97c-a55d5b8b9ed04ca39888b89b82a3f833 on tablet a55d5b8b9ed04ca39888b89b82a3f833 with peer 7845c397230d49358ba2fad9db53f97c
I20240502 20:46:09.091835 8185 consensus_queue.cc:1035] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:09.095906 4637 consensus_queue.cc:237] T dbc1ef85e30d438682bc86935ceefd1b P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } }
W20240502 20:46:09.098047 4573 consensus_peers.cc:485] T dbc1ef85e30d438682bc86935ceefd1b P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867): Couldn't send request to peer a4bf1df7fa6b4948be502c182bfbb833. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dbc1ef85e30d438682bc86935ceefd1b. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:09.098613 4507 raft_consensus.cc:1261] T dbc1ef85e30d438682bc86935ceefd1b P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:09.098614 4374 raft_consensus.cc:1261] T dbc1ef85e30d438682bc86935ceefd1b P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:09.099419 8236 consensus_queue.cc:1035] T dbc1ef85e30d438682bc86935ceefd1b P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:09.099606 6613 consensus_queue.cc:1035] T dbc1ef85e30d438682bc86935ceefd1b P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:09.102185 8168 raft_consensus.cc:2932] T dbc1ef85e30d438682bc86935ceefd1b P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } } }
I20240502 20:46:09.102726 4507 raft_consensus.cc:2932] T dbc1ef85e30d438682bc86935ceefd1b P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } } }
I20240502 20:46:09.105551 4227 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41812:
name: "rebalancer_extra_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:46:09.105909 4233 catalog_manager.cc:5484] T dbc1ef85e30d438682bc86935ceefd1b P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: config changed from index -1 to 4, NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) added. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } } }
I20240502 20:46:09.107053 4374 raft_consensus.cc:2932] T dbc1ef85e30d438682bc86935ceefd1b P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } } }
I20240502 20:46:09.114805 8556 tablet_bootstrap.cc:492] T 5053a7afb4994ecdbc92b129c58903cf P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:09.115123 8556 tablet_bootstrap.cc:492] T 5053a7afb4994ecdbc92b129c58903cf P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap complete.
I20240502 20:46:09.115325 8556 ts_tablet_manager.cc:1367] T 5053a7afb4994ecdbc92b129c58903cf P a4bf1df7fa6b4948be502c182bfbb833: Time spent bootstrapping tablet: real 0.049s user 0.024s sys 0.000s
I20240502 20:46:09.115736 6809 tablet_service.cc:1467] Processing CreateTablet for tablet 52e302882311488ab1cac90ad03fe270 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=553d578c620843f4b11d547972b8d776]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:09.115747 6946 tablet_service.cc:1467] Processing CreateTablet for tablet 52e302882311488ab1cac90ad03fe270 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=553d578c620843f4b11d547972b8d776]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:09.116006 6809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52e302882311488ab1cac90ad03fe270. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:09.116040 6946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52e302882311488ab1cac90ad03fe270. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:09.116669 4354 tablet_service.cc:1467] Processing CreateTablet for tablet ef4002126e8847809add182f262d62d9 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=553d578c620843f4b11d547972b8d776]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:09.116932 4354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef4002126e8847809add182f262d62d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:09.118189 8579 tablet_bootstrap.cc:492] T 52e302882311488ab1cac90ad03fe270 P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap starting.
I20240502 20:46:09.118431 4356 tablet_service.cc:1467] Processing CreateTablet for tablet 93b7f7de1f8c49389c3d8c1e03c67f26 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=553d578c620843f4b11d547972b8d776]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:09.118440 6802 tablet_service.cc:1467] Processing CreateTablet for tablet 93b7f7de1f8c49389c3d8c1e03c67f26 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=553d578c620843f4b11d547972b8d776]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:09.118662 4356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93b7f7de1f8c49389c3d8c1e03c67f26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:09.118671 6802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93b7f7de1f8c49389c3d8c1e03c67f26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:09.119158 8579 tablet_bootstrap.cc:654] T 52e302882311488ab1cac90ad03fe270 P a4bf1df7fa6b4948be502c182bfbb833: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:09.119545 6815 tablet_service.cc:1467] Processing CreateTablet for tablet ef4002126e8847809add182f262d62d9 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=553d578c620843f4b11d547972b8d776]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:09.119771 6815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef4002126e8847809add182f262d62d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:09.120559 8506 raft_consensus.cc:482] T a88bdfa2bca8401d99eec723457a1125 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:09.120664 8506 raft_consensus.cc:504] T a88bdfa2bca8401d99eec723457a1125 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:09.121198 8506 leader_election.cc:288] T a88bdfa2bca8401d99eec723457a1125 P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867), 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:46:09.121203 7079 tablet_service.cc:1467] Processing CreateTablet for tablet 52e302882311488ab1cac90ad03fe270 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=553d578c620843f4b11d547972b8d776]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:09.121491 7079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52e302882311488ab1cac90ad03fe270. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:09.121734 6831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88bdfa2bca8401d99eec723457a1125" candidate_uuid: "7845c397230d49358ba2fad9db53f97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833" is_pre_election: true
I20240502 20:46:09.121855 6831 raft_consensus.cc:2454] T a88bdfa2bca8401d99eec723457a1125 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7845c397230d49358ba2fad9db53f97c in term 0.
I20240502 20:46:09.122005 6946 tablet_service.cc:1467] Processing CreateTablet for tablet ef4002126e8847809add182f262d62d9 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=553d578c620843f4b11d547972b8d776]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:09.122221 6946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef4002126e8847809add182f262d62d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:09.122248 6637 leader_election.cc:302] T a88bdfa2bca8401d99eec723457a1125 P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7845c397230d49358ba2fad9db53f97c, a4bf1df7fa6b4948be502c182bfbb833; no voters:
I20240502 20:46:09.122428 8506 raft_consensus.cc:2781] T a88bdfa2bca8401d99eec723457a1125 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:09.122474 8506 raft_consensus.cc:482] T a88bdfa2bca8401d99eec723457a1125 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:09.122499 8506 raft_consensus.cc:3037] T a88bdfa2bca8401d99eec723457a1125 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:09.123210 8506 raft_consensus.cc:504] T a88bdfa2bca8401d99eec723457a1125 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:09.123507 8506 leader_election.cc:288] T a88bdfa2bca8401d99eec723457a1125 P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 election: Requested vote from peers a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867), 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:46:09.123894 6831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88bdfa2bca8401d99eec723457a1125" candidate_uuid: "7845c397230d49358ba2fad9db53f97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833"
I20240502 20:46:09.123993 6831 raft_consensus.cc:3037] T a88bdfa2bca8401d99eec723457a1125 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:09.124270 6947 tablet_service.cc:1467] Processing CreateTablet for tablet 93b7f7de1f8c49389c3d8c1e03c67f26 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=553d578c620843f4b11d547972b8d776]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:09.124476 6947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93b7f7de1f8c49389c3d8c1e03c67f26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:09.124601 4638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88bdfa2bca8401d99eec723457a1125" candidate_uuid: "7845c397230d49358ba2fad9db53f97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" is_pre_election: true
I20240502 20:46:09.124737 4638 raft_consensus.cc:2454] T a88bdfa2bca8401d99eec723457a1125 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7845c397230d49358ba2fad9db53f97c in term 0.
I20240502 20:46:09.124902 6831 raft_consensus.cc:2454] T a88bdfa2bca8401d99eec723457a1125 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7845c397230d49358ba2fad9db53f97c in term 1.
I20240502 20:46:09.125104 4637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88bdfa2bca8401d99eec723457a1125" candidate_uuid: "7845c397230d49358ba2fad9db53f97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91"
I20240502 20:46:09.125211 4637 raft_consensus.cc:3037] T a88bdfa2bca8401d99eec723457a1125 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:09.125241 6637 leader_election.cc:302] T a88bdfa2bca8401d99eec723457a1125 P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7845c397230d49358ba2fad9db53f97c, a4bf1df7fa6b4948be502c182bfbb833; no voters:
I20240502 20:46:09.125427 8506 raft_consensus.cc:2781] T a88bdfa2bca8401d99eec723457a1125 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:09.125523 8506 raft_consensus.cc:686] T a88bdfa2bca8401d99eec723457a1125 P 7845c397230d49358ba2fad9db53f97c [term 1 LEADER]: Becoming Leader. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Running, Role: LEADER
I20240502 20:46:09.125664 8506 consensus_queue.cc:237] T a88bdfa2bca8401d99eec723457a1125 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:09.126109 4637 raft_consensus.cc:2454] T a88bdfa2bca8401d99eec723457a1125 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7845c397230d49358ba2fad9db53f97c in term 1.
I20240502 20:46:09.127784 4228 catalog_manager.cc:5484] T a88bdfa2bca8401d99eec723457a1125 P 7845c397230d49358ba2fad9db53f97c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7845c397230d49358ba2fad9db53f97c (127.30.32.69). New cstate: current_term: 1 leader_uuid: "7845c397230d49358ba2fad9db53f97c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } health_report { overall_health: HEALTHY } } }
I20240502 20:46:09.128026 8556 raft_consensus.cc:348] T 5053a7afb4994ecdbc92b129c58903cf P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } }
I20240502 20:46:09.128139 8556 raft_consensus.cc:729] T 5053a7afb4994ecdbc92b129c58903cf P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Initialized, Role: LEARNER
I20240502 20:46:09.128243 8556 consensus_queue.cc:260] T 5053a7afb4994ecdbc92b129c58903cf P a4bf1df7fa6b4948be502c182bfbb833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } }
I20240502 20:46:09.128610 8556 ts_tablet_manager.cc:1398] T 5053a7afb4994ecdbc92b129c58903cf P a4bf1df7fa6b4948be502c182bfbb833: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20240502 20:46:09.128942 8681 tablet_bootstrap.cc:492] T 52e302882311488ab1cac90ad03fe270 P a47e5c56f13843018ac584914a8d4707: Bootstrap starting.
I20240502 20:46:09.129122 4528 tablet_copy_service.cc:342] P c3d26eafd51547acb780d4aac5ebe7cc: Request end of tablet copy session a4bf1df7fa6b4948be502c182bfbb833-5053a7afb4994ecdbc92b129c58903cf received from {username='slave'} at 127.30.32.70:46909
I20240502 20:46:09.129199 4528 tablet_copy_service.cc:433] P c3d26eafd51547acb780d4aac5ebe7cc: ending tablet copy session a4bf1df7fa6b4948be502c182bfbb833-5053a7afb4994ecdbc92b129c58903cf on tablet 5053a7afb4994ecdbc92b129c58903cf with peer a4bf1df7fa6b4948be502c182bfbb833
I20240502 20:46:09.131229 8682 tablet_bootstrap.cc:492] T ef4002126e8847809add182f262d62d9 P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:46:09.131832 8684 tablet_bootstrap.cc:492] T 52e302882311488ab1cac90ad03fe270 P 201d2b6485084e7dad2cbb722af94779: Bootstrap starting.
I20240502 20:46:09.132833 8684 tablet_bootstrap.cc:654] T 52e302882311488ab1cac90ad03fe270 P 201d2b6485084e7dad2cbb722af94779: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:09.134335 8682 tablet_bootstrap.cc:654] T ef4002126e8847809add182f262d62d9 P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:09.134459 8684 tablet_bootstrap.cc:492] T 52e302882311488ab1cac90ad03fe270 P 201d2b6485084e7dad2cbb722af94779: No bootstrap required, opened a new log
I20240502 20:46:09.134553 8684 ts_tablet_manager.cc:1367] T 52e302882311488ab1cac90ad03fe270 P 201d2b6485084e7dad2cbb722af94779: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:46:09.134934 8676 ts_tablet_manager.cc:902] T cdf218d5e45b4db19a8e2c22dbc5cc5d P 201d2b6485084e7dad2cbb722af94779: Initiating tablet copy from peer ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:46:09.135053 8684 raft_consensus.cc:348] T 52e302882311488ab1cac90ad03fe270 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:09.135191 8684 raft_consensus.cc:374] T 52e302882311488ab1cac90ad03fe270 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:09.135242 8684 raft_consensus.cc:729] T 52e302882311488ab1cac90ad03fe270 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Initialized, Role: FOLLOWER
I20240502 20:46:09.135330 8676 tablet_copy_client.cc:308] T cdf218d5e45b4db19a8e2c22dbc5cc5d P 201d2b6485084e7dad2cbb722af94779: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.68:38193
I20240502 20:46:09.135421 8684 consensus_queue.cc:260] T 52e302882311488ab1cac90ad03fe270 P 201d2b6485084e7dad2cbb722af94779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:09.135718 8684 ts_tablet_manager.cc:1398] T 52e302882311488ab1cac90ad03fe270 P 201d2b6485084e7dad2cbb722af94779: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:09.135823 8681 tablet_bootstrap.cc:654] T 52e302882311488ab1cac90ad03fe270 P a47e5c56f13843018ac584914a8d4707: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:09.135937 4790 tablet_copy_service.cc:140] P ed47c023452a497192f35d2f617bf9bd: Received BeginTabletCopySession request for tablet cdf218d5e45b4db19a8e2c22dbc5cc5d from peer 201d2b6485084e7dad2cbb722af94779 ({username='slave'} at 127.30.32.72:39947)
I20240502 20:46:09.136041 4790 tablet_copy_service.cc:161] P ed47c023452a497192f35d2f617bf9bd: Beginning new tablet copy session on tablet cdf218d5e45b4db19a8e2c22dbc5cc5d from peer 201d2b6485084e7dad2cbb722af94779 at {username='slave'} at 127.30.32.72:39947: session id = 201d2b6485084e7dad2cbb722af94779-cdf218d5e45b4db19a8e2c22dbc5cc5d
I20240502 20:46:09.136852 8682 tablet_bootstrap.cc:492] T ef4002126e8847809add182f262d62d9 P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:46:09.136943 8682 ts_tablet_manager.cc:1367] T ef4002126e8847809add182f262d62d9 P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 20:46:09.137137 4790 tablet_copy_source_session.cc:205] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:09.137318 8579 tablet_bootstrap.cc:492] T 52e302882311488ab1cac90ad03fe270 P a4bf1df7fa6b4948be502c182bfbb833: No bootstrap required, opened a new log
I20240502 20:46:09.137408 8579 ts_tablet_manager.cc:1367] T 52e302882311488ab1cac90ad03fe270 P a4bf1df7fa6b4948be502c182bfbb833: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.000s
I20240502 20:46:09.137423 8682 raft_consensus.cc:348] T ef4002126e8847809add182f262d62d9 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:09.137523 8682 raft_consensus.cc:374] T ef4002126e8847809add182f262d62d9 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:09.137553 8682 raft_consensus.cc:729] T ef4002126e8847809add182f262d62d9 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:46:09.137665 8682 consensus_queue.cc:260] T ef4002126e8847809add182f262d62d9 P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:09.137867 8579 raft_consensus.cc:348] T 52e302882311488ab1cac90ad03fe270 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:09.137972 8579 raft_consensus.cc:374] T 52e302882311488ab1cac90ad03fe270 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:09.138006 8579 raft_consensus.cc:729] T 52e302882311488ab1cac90ad03fe270 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Initialized, Role: FOLLOWER
I20240502 20:46:09.138121 8579 consensus_queue.cc:260] T 52e302882311488ab1cac90ad03fe270 P a4bf1df7fa6b4948be502c182bfbb833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:09.138427 8579 ts_tablet_manager.cc:1398] T 52e302882311488ab1cac90ad03fe270 P a4bf1df7fa6b4948be502c182bfbb833: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:09.138563 8682 ts_tablet_manager.cc:1398] T ef4002126e8847809add182f262d62d9 P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 20:46:09.138607 8579 tablet_bootstrap.cc:492] T 93b7f7de1f8c49389c3d8c1e03c67f26 P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap starting.
I20240502 20:46:09.138725 8682 tablet_bootstrap.cc:492] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:46:09.139520 8579 tablet_bootstrap.cc:654] T 93b7f7de1f8c49389c3d8c1e03c67f26 P a4bf1df7fa6b4948be502c182bfbb833: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:09.139619 8682 tablet_bootstrap.cc:654] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:09.140650 8579 tablet_bootstrap.cc:492] T 93b7f7de1f8c49389c3d8c1e03c67f26 P a4bf1df7fa6b4948be502c182bfbb833: No bootstrap required, opened a new log
I20240502 20:46:09.141135 8676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdf218d5e45b4db19a8e2c22dbc5cc5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:09.141799 8579 ts_tablet_manager.cc:1367] T 93b7f7de1f8c49389c3d8c1e03c67f26 P a4bf1df7fa6b4948be502c182bfbb833: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 20:46:09.142524 8681 tablet_bootstrap.cc:492] T 52e302882311488ab1cac90ad03fe270 P a47e5c56f13843018ac584914a8d4707: No bootstrap required, opened a new log
I20240502 20:46:09.142671 8681 ts_tablet_manager.cc:1367] T 52e302882311488ab1cac90ad03fe270 P a47e5c56f13843018ac584914a8d4707: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.002s
I20240502 20:46:09.142575 8579 raft_consensus.cc:348] T 93b7f7de1f8c49389c3d8c1e03c67f26 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:09.142828 8579 raft_consensus.cc:374] T 93b7f7de1f8c49389c3d8c1e03c67f26 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:09.142916 8579 raft_consensus.cc:729] T 93b7f7de1f8c49389c3d8c1e03c67f26 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Initialized, Role: FOLLOWER
I20240502 20:46:09.143131 8579 consensus_queue.cc:260] T 93b7f7de1f8c49389c3d8c1e03c67f26 P a4bf1df7fa6b4948be502c182bfbb833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:09.143636 8682 tablet_bootstrap.cc:492] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:46:09.143944 8579 ts_tablet_manager.cc:1398] T 93b7f7de1f8c49389c3d8c1e03c67f26 P a4bf1df7fa6b4948be502c182bfbb833: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 20:46:09.144217 8579 tablet_bootstrap.cc:492] T ef4002126e8847809add182f262d62d9 P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap starting.
I20240502 20:46:09.145161 8682 ts_tablet_manager.cc:1367] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240502 20:46:09.146397 8364 raft_consensus.cc:482] T 52e302882311488ab1cac90ad03fe270 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:09.146472 8364 raft_consensus.cc:504] T 52e302882311488ab1cac90ad03fe270 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:09.146857 8364 leader_election.cc:288] T 52e302882311488ab1cac90ad03fe270 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
I20240502 20:46:09.146970 8676 tablet_copy_client.cc:783] T cdf218d5e45b4db19a8e2c22dbc5cc5d P 201d2b6485084e7dad2cbb722af94779: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:09.147121 8676 tablet_copy_client.cc:647] T cdf218d5e45b4db19a8e2c22dbc5cc5d P 201d2b6485084e7dad2cbb722af94779: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:09.147199 8682 raft_consensus.cc:348] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:09.147310 8682 raft_consensus.cc:374] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:09.147344 8682 raft_consensus.cc:729] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:46:09.147337 6967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52e302882311488ab1cac90ad03fe270" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47e5c56f13843018ac584914a8d4707" is_pre_election: true
I20240502 20:46:09.147783 7099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52e302882311488ab1cac90ad03fe270" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201d2b6485084e7dad2cbb722af94779" is_pre_election: true
I20240502 20:46:09.147478 8682 consensus_queue.cc:260] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:09.147883 8579 tablet_bootstrap.cc:654] T ef4002126e8847809add182f262d62d9 P a4bf1df7fa6b4948be502c182bfbb833: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:09.147912 7099 raft_consensus.cc:2454] T 52e302882311488ab1cac90ad03fe270 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 0.
I20240502 20:46:09.148136 8682 ts_tablet_manager.cc:1398] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 20:46:09.148561 8681 raft_consensus.cc:348] T 52e302882311488ab1cac90ad03fe270 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:09.148713 8681 raft_consensus.cc:374] T 52e302882311488ab1cac90ad03fe270 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:09.148734 8676 tablet_copy_client.cc:523] T cdf218d5e45b4db19a8e2c22dbc5cc5d P 201d2b6485084e7dad2cbb722af94779: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:09.148752 8681 raft_consensus.cc:729] T 52e302882311488ab1cac90ad03fe270 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Initialized, Role: FOLLOWER
I20240502 20:46:09.148890 8681 consensus_queue.cc:260] T 52e302882311488ab1cac90ad03fe270 P a47e5c56f13843018ac584914a8d4707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:09.149235 8681 ts_tablet_manager.cc:1398] T 52e302882311488ab1cac90ad03fe270 P a47e5c56f13843018ac584914a8d4707: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240502 20:46:09.149240 6967 raft_consensus.cc:2454] T 52e302882311488ab1cac90ad03fe270 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 0.
I20240502 20:46:09.149423 8681 tablet_bootstrap.cc:492] T ef4002126e8847809add182f262d62d9 P a47e5c56f13843018ac584914a8d4707: Bootstrap starting.
I20240502 20:46:09.150388 8676 tablet_bootstrap.cc:492] T cdf218d5e45b4db19a8e2c22dbc5cc5d P 201d2b6485084e7dad2cbb722af94779: Bootstrap starting.
I20240502 20:46:09.150465 8681 tablet_bootstrap.cc:654] T ef4002126e8847809add182f262d62d9 P a47e5c56f13843018ac584914a8d4707: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:09.152315 8579 tablet_bootstrap.cc:492] T ef4002126e8847809add182f262d62d9 P a4bf1df7fa6b4948be502c182bfbb833: No bootstrap required, opened a new log
I20240502 20:46:09.152428 8579 ts_tablet_manager.cc:1367] T ef4002126e8847809add182f262d62d9 P a4bf1df7fa6b4948be502c182bfbb833: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20240502 20:46:09.152935 8579 raft_consensus.cc:348] T ef4002126e8847809add182f262d62d9 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:09.153082 8579 raft_consensus.cc:374] T ef4002126e8847809add182f262d62d9 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:09.153121 8579 raft_consensus.cc:729] T ef4002126e8847809add182f262d62d9 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Initialized, Role: FOLLOWER
I20240502 20:46:09.153232 8579 consensus_queue.cc:260] T ef4002126e8847809add182f262d62d9 P a4bf1df7fa6b4948be502c182bfbb833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:09.153496 8579 ts_tablet_manager.cc:1398] T ef4002126e8847809add182f262d62d9 P a4bf1df7fa6b4948be502c182bfbb833: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:09.156524 6768 leader_election.cc:302] T 52e302882311488ab1cac90ad03fe270 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201d2b6485084e7dad2cbb722af94779, a4bf1df7fa6b4948be502c182bfbb833; no voters:
I20240502 20:46:09.156839 8364 raft_consensus.cc:2781] T 52e302882311488ab1cac90ad03fe270 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:09.156894 8364 raft_consensus.cc:482] T 52e302882311488ab1cac90ad03fe270 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:09.157253 8364 raft_consensus.cc:3037] T 52e302882311488ab1cac90ad03fe270 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:09.163683 8364 raft_consensus.cc:504] T 52e302882311488ab1cac90ad03fe270 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:09.164167 8364 leader_election.cc:288] T 52e302882311488ab1cac90ad03fe270 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 election: Requested vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
I20240502 20:46:09.164912 6967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52e302882311488ab1cac90ad03fe270" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47e5c56f13843018ac584914a8d4707"
I20240502 20:46:09.165083 6967 raft_consensus.cc:3037] T 52e302882311488ab1cac90ad03fe270 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:09.165499 8111 consensus_queue.cc:1035] T 63f98288c8b1483493d5018c36e611bb P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:09.166056 7099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52e302882311488ab1cac90ad03fe270" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201d2b6485084e7dad2cbb722af94779"
I20240502 20:46:09.166182 7099 raft_consensus.cc:3037] T 52e302882311488ab1cac90ad03fe270 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:09.166325 6967 raft_consensus.cc:2454] T 52e302882311488ab1cac90ad03fe270 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 1.
I20240502 20:46:09.167197 7099 raft_consensus.cc:2454] T 52e302882311488ab1cac90ad03fe270 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 1.
I20240502 20:46:09.167595 6771 leader_election.cc:302] T 52e302882311488ab1cac90ad03fe270 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a47e5c56f13843018ac584914a8d4707, a4bf1df7fa6b4948be502c182bfbb833; no voters:
I20240502 20:46:09.167851 8364 raft_consensus.cc:2781] T 52e302882311488ab1cac90ad03fe270 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:09.167951 8364 raft_consensus.cc:686] T 52e302882311488ab1cac90ad03fe270 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEADER]: Becoming Leader. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Running, Role: LEADER
I20240502 20:46:09.168090 8364 consensus_queue.cc:237] T 52e302882311488ab1cac90ad03fe270 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:09.170732 4227 catalog_manager.cc:5484] T 52e302882311488ab1cac90ad03fe270 P a4bf1df7fa6b4948be502c182bfbb833 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70). New cstate: current_term: 1 leader_uuid: "a4bf1df7fa6b4948be502c182bfbb833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:09.172122 7996 consensus_queue.cc:1035] T 63f98288c8b1483493d5018c36e611bb P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:09.173549 8681 tablet_bootstrap.cc:492] T ef4002126e8847809add182f262d62d9 P a47e5c56f13843018ac584914a8d4707: No bootstrap required, opened a new log
I20240502 20:46:09.173739 8681 ts_tablet_manager.cc:1367] T ef4002126e8847809add182f262d62d9 P a47e5c56f13843018ac584914a8d4707: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.001s
I20240502 20:46:09.174300 8681 raft_consensus.cc:348] T ef4002126e8847809add182f262d62d9 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:09.174448 8681 raft_consensus.cc:374] T ef4002126e8847809add182f262d62d9 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:09.174815 8681 raft_consensus.cc:729] T ef4002126e8847809add182f262d62d9 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Initialized, Role: FOLLOWER
I20240502 20:46:09.175227 8681 consensus_queue.cc:260] T ef4002126e8847809add182f262d62d9 P a47e5c56f13843018ac584914a8d4707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:09.176399 8681 ts_tablet_manager.cc:1398] T ef4002126e8847809add182f262d62d9 P a47e5c56f13843018ac584914a8d4707: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20240502 20:46:09.177795 8681 tablet_bootstrap.cc:492] T 93b7f7de1f8c49389c3d8c1e03c67f26 P a47e5c56f13843018ac584914a8d4707: Bootstrap starting.
I20240502 20:46:09.179483 8681 tablet_bootstrap.cc:654] T 93b7f7de1f8c49389c3d8c1e03c67f26 P a47e5c56f13843018ac584914a8d4707: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:09.183938 8681 tablet_bootstrap.cc:492] T 93b7f7de1f8c49389c3d8c1e03c67f26 P a47e5c56f13843018ac584914a8d4707: No bootstrap required, opened a new log
I20240502 20:46:09.184047 8681 ts_tablet_manager.cc:1367] T 93b7f7de1f8c49389c3d8c1e03c67f26 P a47e5c56f13843018ac584914a8d4707: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240502 20:46:09.184551 8681 raft_consensus.cc:348] T 93b7f7de1f8c49389c3d8c1e03c67f26 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:09.184700 8681 raft_consensus.cc:374] T 93b7f7de1f8c49389c3d8c1e03c67f26 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:09.184762 8681 raft_consensus.cc:729] T 93b7f7de1f8c49389c3d8c1e03c67f26 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Initialized, Role: FOLLOWER
I20240502 20:46:09.184931 8681 consensus_queue.cc:260] T 93b7f7de1f8c49389c3d8c1e03c67f26 P a47e5c56f13843018ac584914a8d4707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:09.185245 8681 ts_tablet_manager.cc:1398] T 93b7f7de1f8c49389c3d8c1e03c67f26 P a47e5c56f13843018ac584914a8d4707: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 20:46:09.188889 8676 tablet_bootstrap.cc:492] T cdf218d5e45b4db19a8e2c22dbc5cc5d P 201d2b6485084e7dad2cbb722af94779: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:09.189190 8676 tablet_bootstrap.cc:492] T cdf218d5e45b4db19a8e2c22dbc5cc5d P 201d2b6485084e7dad2cbb722af94779: Bootstrap complete.
I20240502 20:46:09.189373 8676 ts_tablet_manager.cc:1367] T cdf218d5e45b4db19a8e2c22dbc5cc5d P 201d2b6485084e7dad2cbb722af94779: Time spent bootstrapping tablet: real 0.039s user 0.026s sys 0.003s
I20240502 20:46:09.189981 8676 raft_consensus.cc:348] T cdf218d5e45b4db19a8e2c22dbc5cc5d P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } }
I20240502 20:46:09.190238 8676 raft_consensus.cc:729] T cdf218d5e45b4db19a8e2c22dbc5cc5d P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Initialized, Role: LEARNER
I20240502 20:46:09.190409 8676 consensus_queue.cc:260] T cdf218d5e45b4db19a8e2c22dbc5cc5d P 201d2b6485084e7dad2cbb722af94779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } }
I20240502 20:46:09.190794 8676 ts_tablet_manager.cc:1398] T cdf218d5e45b4db19a8e2c22dbc5cc5d P 201d2b6485084e7dad2cbb722af94779: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:09.191428 4790 tablet_copy_service.cc:342] P ed47c023452a497192f35d2f617bf9bd: Request end of tablet copy session 201d2b6485084e7dad2cbb722af94779-cdf218d5e45b4db19a8e2c22dbc5cc5d received from {username='slave'} at 127.30.32.72:39947
I20240502 20:46:09.191510 4790 tablet_copy_service.cc:433] P ed47c023452a497192f35d2f617bf9bd: ending tablet copy session 201d2b6485084e7dad2cbb722af94779-cdf218d5e45b4db19a8e2c22dbc5cc5d on tablet cdf218d5e45b4db19a8e2c22dbc5cc5d with peer 201d2b6485084e7dad2cbb722af94779
I20240502 20:46:09.239511 6967 raft_consensus.cc:1203] T 66b8f583dfd24c2aa5a59457a8e58fbb P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:09.264804 4230 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 20:46:09.269954 4637 raft_consensus.cc:1261] T a88bdfa2bca8401d99eec723457a1125 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer 7845c397230d49358ba2fad9db53f97c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:09.270396 6831 raft_consensus.cc:1261] T a88bdfa2bca8401d99eec723457a1125 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Refusing update from remote peer 7845c397230d49358ba2fad9db53f97c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:09.270802 8505 consensus_queue.cc:1035] T a88bdfa2bca8401d99eec723457a1125 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:09.271667 8505 consensus_queue.cc:1035] T a88bdfa2bca8401d99eec723457a1125 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:09.274816 4227 catalog_manager.cc:5576] TS a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867) does not have the latest schema for tablet a88bdfa2bca8401d99eec723457a1125 (table rebalancer_dynamic_table [id=9a174cd46c574a628620ba4cba9d6466]). Expected version 1 got 0
I20240502 20:46:09.275735 8777 tablet.cc:1722] T a88bdfa2bca8401d99eec723457a1125 P a4bf1df7fa6b4948be502c182bfbb833: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 20:46:09.276119 8777 tablet.cc:1620] T a88bdfa2bca8401d99eec723457a1125 P a4bf1df7fa6b4948be502c182bfbb833: MemRowSet was empty: no flush needed.
I20240502 20:46:09.276458 4228 catalog_manager.cc:5576] TS 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141) does not have the latest schema for tablet a88bdfa2bca8401d99eec723457a1125 (table rebalancer_dynamic_table [id=9a174cd46c574a628620ba4cba9d6466]). Expected version 1 got 0
I20240502 20:46:09.276836 4227 catalog_manager.cc:5576] TS 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429) does not have the latest schema for tablet a88bdfa2bca8401d99eec723457a1125 (table rebalancer_dynamic_table [id=9a174cd46c574a628620ba4cba9d6466]). Expected version 1 got 0
I20240502 20:46:09.277536 8776 tablet.cc:1722] T a88bdfa2bca8401d99eec723457a1125 P 7845c397230d49358ba2fad9db53f97c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 20:46:09.278064 8776 tablet.cc:1620] T a88bdfa2bca8401d99eec723457a1125 P 7845c397230d49358ba2fad9db53f97c: MemRowSet was empty: no flush needed.
I20240502 20:46:09.280921 8778 tablet.cc:1722] T a88bdfa2bca8401d99eec723457a1125 P 923cfab926e7443f8ffd7b21ad0a9d91: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 20:46:09.281764 4230 catalog_manager.cc:5576] TS 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141) does not have the latest schema for tablet a88bdfa2bca8401d99eec723457a1125 (table rebalancer_dynamic_table [id=9a174cd46c574a628620ba4cba9d6466]). Expected version 1 got 0
I20240502 20:46:09.282740 8778 tablet.cc:1620] T a88bdfa2bca8401d99eec723457a1125 P 923cfab926e7443f8ffd7b21ad0a9d91: MemRowSet was empty: no flush needed.
I20240502 20:46:09.282362 4227 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: rebalancer_dynamic_table [id=9a174cd46c574a628620ba4cba9d6466] alter complete (version 1)
I20240502 20:46:09.285320 8775 ts_tablet_manager.cc:902] T 747e7331d7ad4dd4b132672254037e1b P c3d26eafd51547acb780d4aac5ebe7cc: Initiating tablet copy from peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
I20240502 20:46:09.286520 8556 ts_tablet_manager.cc:902] T 45ade8c672d8445eaffb179772e3759a P a4bf1df7fa6b4948be502c182bfbb833: Initiating tablet copy from peer c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097)
I20240502 20:46:09.287062 8556 tablet_copy_client.cc:308] T 45ade8c672d8445eaffb179772e3759a P a4bf1df7fa6b4948be502c182bfbb833: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.66:42097
I20240502 20:46:09.287576 4528 tablet_copy_service.cc:140] P c3d26eafd51547acb780d4aac5ebe7cc: Received BeginTabletCopySession request for tablet 45ade8c672d8445eaffb179772e3759a from peer a4bf1df7fa6b4948be502c182bfbb833 ({username='slave'} at 127.30.32.70:46909)
I20240502 20:46:09.287683 4528 tablet_copy_service.cc:161] P c3d26eafd51547acb780d4aac5ebe7cc: Beginning new tablet copy session on tablet 45ade8c672d8445eaffb179772e3759a from peer a4bf1df7fa6b4948be502c182bfbb833 at {username='slave'} at 127.30.32.70:46909: session id = a4bf1df7fa6b4948be502c182bfbb833-45ade8c672d8445eaffb179772e3759a
I20240502 20:46:09.287832 8775 tablet_copy_client.cc:308] T 747e7331d7ad4dd4b132672254037e1b P c3d26eafd51547acb780d4aac5ebe7cc: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.71:46841
I20240502 20:46:09.288761 4528 tablet_copy_source_session.cc:205] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:09.288870 6987 tablet_copy_service.cc:140] P a47e5c56f13843018ac584914a8d4707: Received BeginTabletCopySession request for tablet 747e7331d7ad4dd4b132672254037e1b from peer c3d26eafd51547acb780d4aac5ebe7cc ({username='slave'} at 127.30.32.66:49749)
I20240502 20:46:09.289047 6987 tablet_copy_service.cc:161] P a47e5c56f13843018ac584914a8d4707: Beginning new tablet copy session on tablet 747e7331d7ad4dd4b132672254037e1b from peer c3d26eafd51547acb780d4aac5ebe7cc at {username='slave'} at 127.30.32.66:49749: session id = c3d26eafd51547acb780d4aac5ebe7cc-747e7331d7ad4dd4b132672254037e1b
I20240502 20:46:09.289829 8556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45ade8c672d8445eaffb179772e3759a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:09.290483 6987 tablet_copy_source_session.cc:205] T 747e7331d7ad4dd4b132672254037e1b P a47e5c56f13843018ac584914a8d4707: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:09.292035 8556 tablet_copy_client.cc:783] T 45ade8c672d8445eaffb179772e3759a P a4bf1df7fa6b4948be502c182bfbb833: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:09.292171 8556 tablet_copy_client.cc:647] T 45ade8c672d8445eaffb179772e3759a P a4bf1df7fa6b4948be502c182bfbb833: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:09.292610 8775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 747e7331d7ad4dd4b132672254037e1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:09.293485 8556 tablet_copy_client.cc:523] T 45ade8c672d8445eaffb179772e3759a P a4bf1df7fa6b4948be502c182bfbb833: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:09.294489 8556 tablet_bootstrap.cc:492] T 45ade8c672d8445eaffb179772e3759a P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap starting.
I20240502 20:46:09.294791 8775 tablet_copy_client.cc:783] T 747e7331d7ad4dd4b132672254037e1b P c3d26eafd51547acb780d4aac5ebe7cc: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:09.294975 8775 tablet_copy_client.cc:647] T 747e7331d7ad4dd4b132672254037e1b P c3d26eafd51547acb780d4aac5ebe7cc: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:09.296672 8676 ts_tablet_manager.cc:902] T e5a4dc24914c4f37a14481b77c4fcdaa P 201d2b6485084e7dad2cbb722af94779: Initiating tablet copy from peer ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:46:09.296707 8775 tablet_copy_client.cc:523] T 747e7331d7ad4dd4b132672254037e1b P c3d26eafd51547acb780d4aac5ebe7cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:09.297166 8676 tablet_copy_client.cc:308] T e5a4dc24914c4f37a14481b77c4fcdaa P 201d2b6485084e7dad2cbb722af94779: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.68:38193
I20240502 20:46:09.297691 4790 tablet_copy_service.cc:140] P ed47c023452a497192f35d2f617bf9bd: Received BeginTabletCopySession request for tablet e5a4dc24914c4f37a14481b77c4fcdaa from peer 201d2b6485084e7dad2cbb722af94779 ({username='slave'} at 127.30.32.72:39947)
I20240502 20:46:09.297816 4790 tablet_copy_service.cc:161] P ed47c023452a497192f35d2f617bf9bd: Beginning new tablet copy session on tablet e5a4dc24914c4f37a14481b77c4fcdaa from peer 201d2b6485084e7dad2cbb722af94779 at {username='slave'} at 127.30.32.72:39947: session id = 201d2b6485084e7dad2cbb722af94779-e5a4dc24914c4f37a14481b77c4fcdaa
I20240502 20:46:09.298808 8775 tablet_bootstrap.cc:492] T 747e7331d7ad4dd4b132672254037e1b P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:46:09.298941 4790 tablet_copy_source_session.cc:205] T e5a4dc24914c4f37a14481b77c4fcdaa P ed47c023452a497192f35d2f617bf9bd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:09.300035 8676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5a4dc24914c4f37a14481b77c4fcdaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:09.302345 8676 tablet_copy_client.cc:783] T e5a4dc24914c4f37a14481b77c4fcdaa P 201d2b6485084e7dad2cbb722af94779: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:09.302501 8676 tablet_copy_client.cc:647] T e5a4dc24914c4f37a14481b77c4fcdaa P 201d2b6485084e7dad2cbb722af94779: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:09.303818 8676 tablet_copy_client.cc:523] T e5a4dc24914c4f37a14481b77c4fcdaa P 201d2b6485084e7dad2cbb722af94779: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:09.305166 8676 tablet_bootstrap.cc:492] T e5a4dc24914c4f37a14481b77c4fcdaa P 201d2b6485084e7dad2cbb722af94779: Bootstrap starting.
I20240502 20:46:09.305312 8775 tablet_bootstrap.cc:492] T 747e7331d7ad4dd4b132672254037e1b P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:09.305578 8775 tablet_bootstrap.cc:492] T 747e7331d7ad4dd4b132672254037e1b P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap complete.
I20240502 20:46:09.305732 8775 ts_tablet_manager.cc:1367] T 747e7331d7ad4dd4b132672254037e1b P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.001s
I20240502 20:46:09.306197 8775 raft_consensus.cc:348] T 747e7331d7ad4dd4b132672254037e1b P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } }
I20240502 20:46:09.306331 8775 raft_consensus.cc:729] T 747e7331d7ad4dd4b132672254037e1b P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: LEARNER
I20240502 20:46:09.306444 8775 consensus_queue.cc:260] T 747e7331d7ad4dd4b132672254037e1b P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } }
I20240502 20:46:09.306840 8775 ts_tablet_manager.cc:1398] T 747e7331d7ad4dd4b132672254037e1b P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 20:46:09.308573 8528 raft_consensus.cc:482] T ef4002126e8847809add182f262d62d9 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:09.308686 8528 raft_consensus.cc:504] T ef4002126e8847809add182f262d62d9 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:09.309227 8528 leader_election.cc:288] T ef4002126e8847809add182f262d62d9 P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867), c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:46:09.309480 6987 tablet_copy_service.cc:342] P a47e5c56f13843018ac584914a8d4707: Request end of tablet copy session c3d26eafd51547acb780d4aac5ebe7cc-747e7331d7ad4dd4b132672254037e1b received from {username='slave'} at 127.30.32.66:49749
I20240502 20:46:09.309579 6987 tablet_copy_service.cc:433] P a47e5c56f13843018ac584914a8d4707: ending tablet copy session c3d26eafd51547acb780d4aac5ebe7cc-747e7331d7ad4dd4b132672254037e1b on tablet 747e7331d7ad4dd4b132672254037e1b with peer c3d26eafd51547acb780d4aac5ebe7cc
I20240502 20:46:09.311039 6831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef4002126e8847809add182f262d62d9" candidate_uuid: "a47e5c56f13843018ac584914a8d4707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833" is_pre_election: true
I20240502 20:46:09.311192 6831 raft_consensus.cc:2454] T ef4002126e8847809add182f262d62d9 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a47e5c56f13843018ac584914a8d4707 in term 0.
I20240502 20:46:09.311522 6901 leader_election.cc:302] T ef4002126e8847809add182f262d62d9 P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a47e5c56f13843018ac584914a8d4707, a4bf1df7fa6b4948be502c182bfbb833; no voters:
I20240502 20:46:09.312032 4374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef4002126e8847809add182f262d62d9" candidate_uuid: "a47e5c56f13843018ac584914a8d4707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2" is_pre_election: true
I20240502 20:46:09.312191 4374 raft_consensus.cc:2454] T ef4002126e8847809add182f262d62d9 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a47e5c56f13843018ac584914a8d4707 in term 0.
I20240502 20:46:09.312963 8528 raft_consensus.cc:2781] T ef4002126e8847809add182f262d62d9 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:09.313076 8528 raft_consensus.cc:482] T ef4002126e8847809add182f262d62d9 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:09.313133 8528 raft_consensus.cc:3037] T ef4002126e8847809add182f262d62d9 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:09.314105 8528 raft_consensus.cc:504] T ef4002126e8847809add182f262d62d9 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:09.314508 8528 leader_election.cc:288] T ef4002126e8847809add182f262d62d9 P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 1 election: Requested vote from peers a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867), c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:46:09.314924 4374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef4002126e8847809add182f262d62d9" candidate_uuid: "a47e5c56f13843018ac584914a8d4707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2"
I20240502 20:46:09.314924 6831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef4002126e8847809add182f262d62d9" candidate_uuid: "a47e5c56f13843018ac584914a8d4707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833"
I20240502 20:46:09.315044 6831 raft_consensus.cc:3037] T ef4002126e8847809add182f262d62d9 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:09.315044 4374 raft_consensus.cc:3037] T ef4002126e8847809add182f262d62d9 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:09.316053 6831 raft_consensus.cc:2454] T ef4002126e8847809add182f262d62d9 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a47e5c56f13843018ac584914a8d4707 in term 1.
I20240502 20:46:09.316049 4374 raft_consensus.cc:2454] T ef4002126e8847809add182f262d62d9 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a47e5c56f13843018ac584914a8d4707 in term 1.
I20240502 20:46:09.316633 6901 leader_election.cc:302] T ef4002126e8847809add182f262d62d9 P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a47e5c56f13843018ac584914a8d4707, a4bf1df7fa6b4948be502c182bfbb833; no voters:
I20240502 20:46:09.318189 8528 raft_consensus.cc:2781] T ef4002126e8847809add182f262d62d9 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:09.318289 8528 raft_consensus.cc:686] T ef4002126e8847809add182f262d62d9 P a47e5c56f13843018ac584914a8d4707 [term 1 LEADER]: Becoming Leader. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Running, Role: LEADER
I20240502 20:46:09.318446 8528 consensus_queue.cc:237] T ef4002126e8847809add182f262d62d9 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:09.320364 4230 catalog_manager.cc:5484] T ef4002126e8847809add182f262d62d9 P a47e5c56f13843018ac584914a8d4707 reported cstate change: term changed from 0 to 1, leader changed from <none> to a47e5c56f13843018ac584914a8d4707 (127.30.32.71). New cstate: current_term: 1 leader_uuid: "a47e5c56f13843018ac584914a8d4707" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:09.324072 8556 tablet_bootstrap.cc:492] T 45ade8c672d8445eaffb179772e3759a P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:09.324391 8556 tablet_bootstrap.cc:492] T 45ade8c672d8445eaffb179772e3759a P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap complete.
I20240502 20:46:09.324589 8556 ts_tablet_manager.cc:1367] T 45ade8c672d8445eaffb179772e3759a P a4bf1df7fa6b4948be502c182bfbb833: Time spent bootstrapping tablet: real 0.030s user 0.024s sys 0.000s
I20240502 20:46:09.325145 8556 raft_consensus.cc:348] T 45ade8c672d8445eaffb179772e3759a P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } }
I20240502 20:46:09.325273 8556 raft_consensus.cc:729] T 45ade8c672d8445eaffb179772e3759a P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Initialized, Role: LEARNER
I20240502 20:46:09.325382 8556 consensus_queue.cc:260] T 45ade8c672d8445eaffb179772e3759a P a4bf1df7fa6b4948be502c182bfbb833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } }
I20240502 20:46:09.325670 8556 ts_tablet_manager.cc:1398] T 45ade8c672d8445eaffb179772e3759a P a4bf1df7fa6b4948be502c182bfbb833: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240502 20:46:09.325678 6702 raft_consensus.cc:1203] T d6582234196c4fe4a82b3182787700de P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:09.326395 4528 tablet_copy_service.cc:342] P c3d26eafd51547acb780d4aac5ebe7cc: Request end of tablet copy session a4bf1df7fa6b4948be502c182bfbb833-45ade8c672d8445eaffb179772e3759a received from {username='slave'} at 127.30.32.70:46909
I20240502 20:46:09.326483 4528 tablet_copy_service.cc:433] P c3d26eafd51547acb780d4aac5ebe7cc: ending tablet copy session a4bf1df7fa6b4948be502c182bfbb833-45ade8c672d8445eaffb179772e3759a on tablet 45ade8c672d8445eaffb179772e3759a with peer a4bf1df7fa6b4948be502c182bfbb833
I20240502 20:46:09.326488 4227 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240502 20:46:09.326606 4227 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240502 20:46:09.330121 4227 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: Sending DeleteTablet for 4 replicas of tablet e5a4dc24914c4f37a14481b77c4fcdaa
I20240502 20:46:09.330536 4227 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: Sending DeleteTablet for 3 replicas of tablet 6c9adf5004a649979b9abb97df5c0c28
I20240502 20:46:09.330680 4487 tablet_service.cc:1514] Processing DeleteTablet for tablet e5a4dc24914c4f37a14481b77c4fcdaa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:09 UTC) from {username='slave'} at 127.0.0.1:48890
I20240502 20:46:09.330816 4227 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: Sending DeleteTablet for 4 replicas of tablet 04c72d7d63574dea99cb2f7c14199a73
I20240502 20:46:09.330930 4750 tablet_service.cc:1514] Processing DeleteTablet for tablet e5a4dc24914c4f37a14481b77c4fcdaa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:09 UTC) from {username='slave'} at 127.0.0.1:38526
I20240502 20:46:09.330964 7079 tablet_service.cc:1514] Processing DeleteTablet for tablet e5a4dc24914c4f37a14481b77c4fcdaa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:09 UTC) from {username='slave'} at 127.0.0.1:59162
I20240502 20:46:09.331136 8141 tablet_replica.cc:330] T e5a4dc24914c4f37a14481b77c4fcdaa P c3d26eafd51547acb780d4aac5ebe7cc: stopping tablet replica
I20240502 20:46:09.331297 8141 raft_consensus.cc:2229] T e5a4dc24914c4f37a14481b77c4fcdaa P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:09.331379 4488 tablet_service.cc:1514] Processing DeleteTablet for tablet 04c72d7d63574dea99cb2f7c14199a73 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:09 UTC) from {username='slave'} at 127.0.0.1:48890
I20240502 20:46:09.331406 6682 tablet_service.cc:1514] Processing DeleteTablet for tablet 04c72d7d63574dea99cb2f7c14199a73 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:09 UTC) from {username='slave'} at 127.0.0.1:37506
I20240502 20:46:09.331475 8141 raft_consensus.cc:2258] T e5a4dc24914c4f37a14481b77c4fcdaa P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:09.331670 7299 tablet_replica.cc:330] T e5a4dc24914c4f37a14481b77c4fcdaa P ed47c023452a497192f35d2f617bf9bd: stopping tablet replica
I20240502 20:46:09.331815 7299 raft_consensus.cc:2229] T e5a4dc24914c4f37a14481b77c4fcdaa P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:09.331986 4618 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c9adf5004a649979b9abb97df5c0c28 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:09 UTC) from {username='slave'} at 127.0.0.1:57068
I20240502 20:46:09.332114 7299 raft_consensus.cc:2258] T e5a4dc24914c4f37a14481b77c4fcdaa P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:09.332211 8510 tablet_replica.cc:330] T 6c9adf5004a649979b9abb97df5c0c28 P 923cfab926e7443f8ffd7b21ad0a9d91: stopping tablet replica
I20240502 20:46:09.332312 8141 ts_tablet_manager.cc:1875] T e5a4dc24914c4f37a14481b77c4fcdaa P c3d26eafd51547acb780d4aac5ebe7cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:09.332350 8510 raft_consensus.cc:2229] T 6c9adf5004a649979b9abb97df5c0c28 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:09.332424 4487 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c9adf5004a649979b9abb97df5c0c28 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:09 UTC) from {username='slave'} at 127.0.0.1:48890
I20240502 20:46:09.332489 8510 raft_consensus.cc:2258] T 6c9adf5004a649979b9abb97df5c0c28 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:09.332969 7299 ts_tablet_manager.cc:1875] T e5a4dc24914c4f37a14481b77c4fcdaa P ed47c023452a497192f35d2f617bf9bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:09.333094 8510 ts_tablet_manager.cc:1875] T 6c9adf5004a649979b9abb97df5c0c28 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:09.334833 8141 ts_tablet_manager.cc:1888] T e5a4dc24914c4f37a14481b77c4fcdaa P c3d26eafd51547acb780d4aac5ebe7cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:09.334901 8141 log.cc:1196] T e5a4dc24914c4f37a14481b77c4fcdaa P c3d26eafd51547acb780d4aac5ebe7cc: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal/wals/e5a4dc24914c4f37a14481b77c4fcdaa
I20240502 20:46:09.335203 8141 ts_tablet_manager.cc:1909] T e5a4dc24914c4f37a14481b77c4fcdaa P c3d26eafd51547acb780d4aac5ebe7cc: Deleting consensus metadata
I20240502 20:46:09.335441 7299 ts_tablet_manager.cc:1888] T e5a4dc24914c4f37a14481b77c4fcdaa P ed47c023452a497192f35d2f617bf9bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:09.335517 7299 log.cc:1196] T e5a4dc24914c4f37a14481b77c4fcdaa P ed47c023452a497192f35d2f617bf9bd: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal/wals/e5a4dc24914c4f37a14481b77c4fcdaa
I20240502 20:46:09.335506 8510 ts_tablet_manager.cc:1888] T 6c9adf5004a649979b9abb97df5c0c28 P 923cfab926e7443f8ffd7b21ad0a9d91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 20:46:09.335573 8510 log.cc:1196] T 6c9adf5004a649979b9abb97df5c0c28 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal/wals/6c9adf5004a649979b9abb97df5c0c28
I20240502 20:46:09.335753 7299 ts_tablet_manager.cc:1909] T e5a4dc24914c4f37a14481b77c4fcdaa P ed47c023452a497192f35d2f617bf9bd: Deleting consensus metadata
I20240502 20:46:09.335839 8510 ts_tablet_manager.cc:1909] T 6c9adf5004a649979b9abb97df5c0c28 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting consensus metadata
I20240502 20:46:09.335953 8141 tablet_replica.cc:330] T 04c72d7d63574dea99cb2f7c14199a73 P c3d26eafd51547acb780d4aac5ebe7cc: stopping tablet replica
I20240502 20:46:09.336072 8141 raft_consensus.cc:2229] T 04c72d7d63574dea99cb2f7c14199a73 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:09.336071 4221 catalog_manager.cc:4830] TS c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097): tablet e5a4dc24914c4f37a14481b77c4fcdaa (table rebalance_test_table_14 [id=a699ad9359e6422b8223c2844e565c67]) successfully deleted
I20240502 20:46:09.336207 8141 raft_consensus.cc:2258] T 04c72d7d63574dea99cb2f7c14199a73 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:09.336301 6702 raft_consensus.cc:1203] T 180902058815460189f75de4704d1e84 P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 20:46:09.336524 8806 tablet_replica.cc:330] T 04c72d7d63574dea99cb2f7c14199a73 P 7845c397230d49358ba2fad9db53f97c: stopping tablet replica
I20240502 20:46:09.336586 4219 catalog_manager.cc:4830] TS 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141): tablet 6c9adf5004a649979b9abb97df5c0c28 (table rebalance_test_table_14 [id=a699ad9359e6422b8223c2844e565c67]) successfully deleted
I20240502 20:46:09.336690 8806 raft_consensus.cc:2229] T 04c72d7d63574dea99cb2f7c14199a73 P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Raft consensus shutting down.
I20240502 20:46:09.336803 8806 raft_consensus.cc:2258] T 04c72d7d63574dea99cb2f7c14199a73 P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Raft consensus is shut down!
I20240502 20:46:09.337107 4749 tablet_service.cc:1514] Processing DeleteTablet for tablet 04c72d7d63574dea99cb2f7c14199a73 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:09 UTC) from {username='slave'} at 127.0.0.1:38526
I20240502 20:46:09.337287 8141 ts_tablet_manager.cc:1875] T 04c72d7d63574dea99cb2f7c14199a73 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:09.337313 4356 tablet_service.cc:1514] Processing DeleteTablet for tablet e5a4dc24914c4f37a14481b77c4fcdaa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:09 UTC) from {username='slave'} at 127.0.0.1:38306
I20240502 20:46:09.337394 4354 tablet_service.cc:1514] Processing DeleteTablet for tablet 04c72d7d63574dea99cb2f7c14199a73 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:09 UTC) from {username='slave'} at 127.0.0.1:38306
I20240502 20:46:09.337630 7297 tablet_replica.cc:330] T e5a4dc24914c4f37a14481b77c4fcdaa P c92e6e46782849479c8a0c6d82276fb2: stopping tablet replica
I20240502 20:46:09.337776 7297 raft_consensus.cc:2229] T e5a4dc24914c4f37a14481b77c4fcdaa P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:09.337880 8806 ts_tablet_manager.cc:1875] T 04c72d7d63574dea99cb2f7c14199a73 P 7845c397230d49358ba2fad9db53f97c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:09.337929 7297 raft_consensus.cc:2258] T e5a4dc24914c4f37a14481b77c4fcdaa P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:09.338368 7937 raft_consensus.cc:482] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:09.338450 7937 raft_consensus.cc:504] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:09.338649 7297 ts_tablet_manager.cc:1875] T e5a4dc24914c4f37a14481b77c4fcdaa P c92e6e46782849479c8a0c6d82276fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:09.338965 7937 leader_election.cc:288] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867)
I20240502 20:46:09.339326 8141 ts_tablet_manager.cc:1888] T 04c72d7d63574dea99cb2f7c14199a73 P c3d26eafd51547acb780d4aac5ebe7cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:09.339390 8141 log.cc:1196] T 04c72d7d63574dea99cb2f7c14199a73 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal/wals/04c72d7d63574dea99cb2f7c14199a73
I20240502 20:46:09.339664 8141 ts_tablet_manager.cc:1909] T 04c72d7d63574dea99cb2f7c14199a73 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting consensus metadata
I20240502 20:46:09.339855 6967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b7f7de1f8c49389c3d8c1e03c67f26" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47e5c56f13843018ac584914a8d4707" is_pre_election: true
I20240502 20:46:09.340005 6967 raft_consensus.cc:2454] T 93b7f7de1f8c49389c3d8c1e03c67f26 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 0.
I20240502 20:46:09.340216 8141 tablet_replica.cc:330] T 6c9adf5004a649979b9abb97df5c0c28 P c3d26eafd51547acb780d4aac5ebe7cc: stopping tablet replica
I20240502 20:46:09.340355 8141 raft_consensus.cc:2229] T 6c9adf5004a649979b9abb97df5c0c28 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:09.340351 4313 leader_election.cc:302] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a47e5c56f13843018ac584914a8d4707, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:46:09.340502 8141 raft_consensus.cc:2258] T 6c9adf5004a649979b9abb97df5c0c28 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:09.340543 7937 raft_consensus.cc:2781] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:09.340593 7937 raft_consensus.cc:482] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:09.340636 7937 raft_consensus.cc:3037] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:09.340847 8806 ts_tablet_manager.cc:1888] T 04c72d7d63574dea99cb2f7c14199a73 P 7845c397230d49358ba2fad9db53f97c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:09.340916 8806 log.cc:1196] T 04c72d7d63574dea99cb2f7c14199a73 P 7845c397230d49358ba2fad9db53f97c: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/wal/wals/04c72d7d63574dea99cb2f7c14199a73
I20240502 20:46:09.340905 7297 ts_tablet_manager.cc:1888] T e5a4dc24914c4f37a14481b77c4fcdaa P c92e6e46782849479c8a0c6d82276fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:09.341224 8806 ts_tablet_manager.cc:1909] T 04c72d7d63574dea99cb2f7c14199a73 P 7845c397230d49358ba2fad9db53f97c: Deleting consensus metadata
I20240502 20:46:09.341620 4750 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c9adf5004a649979b9abb97df5c0c28 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:09 UTC) from {username='slave'} at 127.0.0.1:38526
I20240502 20:46:09.342257 8141 ts_tablet_manager.cc:1875] T 6c9adf5004a649979b9abb97df5c0c28 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:09.343322 4221 catalog_manager.cc:4830] TS 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): tablet 04c72d7d63574dea99cb2f7c14199a73 (table rebalance_test_table_14 [id=a699ad9359e6422b8223c2844e565c67]) successfully deleted
I20240502 20:46:09.343504 4221 catalog_manager.cc:4830] TS c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097): tablet 04c72d7d63574dea99cb2f7c14199a73 (table rebalance_test_table_14 [id=a699ad9359e6422b8223c2844e565c67]) successfully deleted
I20240502 20:46:09.344099 8141 ts_tablet_manager.cc:1888] T 6c9adf5004a649979b9abb97df5c0c28 P c3d26eafd51547acb780d4aac5ebe7cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 20:46:09.344177 8141 log.cc:1196] T 6c9adf5004a649979b9abb97df5c0c28 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal/wals/6c9adf5004a649979b9abb97df5c0c28
I20240502 20:46:09.340970 7297 log.cc:1196] T e5a4dc24914c4f37a14481b77c4fcdaa P c92e6e46782849479c8a0c6d82276fb2: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal/wals/e5a4dc24914c4f37a14481b77c4fcdaa
I20240502 20:46:09.341658 7937 raft_consensus.cc:504] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:09.344460 8141 ts_tablet_manager.cc:1909] T 6c9adf5004a649979b9abb97df5c0c28 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting consensus metadata
I20240502 20:46:09.344569 7297 ts_tablet_manager.cc:1909] T e5a4dc24914c4f37a14481b77c4fcdaa P c92e6e46782849479c8a0c6d82276fb2: Deleting consensus metadata
I20240502 20:46:09.344750 7937 leader_election.cc:288] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 election: Requested vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867)
I20240502 20:46:09.345149 7297 tablet_replica.cc:330] T 04c72d7d63574dea99cb2f7c14199a73 P c92e6e46782849479c8a0c6d82276fb2: stopping tablet replica
I20240502 20:46:09.345260 7297 raft_consensus.cc:2229] T 04c72d7d63574dea99cb2f7c14199a73 P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:09.345264 6967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b7f7de1f8c49389c3d8c1e03c67f26" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47e5c56f13843018ac584914a8d4707"
I20240502 20:46:09.345363 6967 raft_consensus.cc:3037] T 93b7f7de1f8c49389c3d8c1e03c67f26 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:09.345471 7297 raft_consensus.cc:2258] T 04c72d7d63574dea99cb2f7c14199a73 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:09.346172 7297 ts_tablet_manager.cc:1875] T 04c72d7d63574dea99cb2f7c14199a73 P c92e6e46782849479c8a0c6d82276fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:09.346352 6967 raft_consensus.cc:2454] T 93b7f7de1f8c49389c3d8c1e03c67f26 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 1.
I20240502 20:46:09.346583 4222 catalog_manager.cc:4830] TS c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827): tablet e5a4dc24914c4f37a14481b77c4fcdaa (table rebalance_test_table_14 [id=a699ad9359e6422b8223c2844e565c67]) successfully deleted
I20240502 20:46:09.346926 4313 leader_election.cc:302] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a47e5c56f13843018ac584914a8d4707, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:46:09.347079 7937 raft_consensus.cc:2781] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:09.347096 4221 catalog_manager.cc:4830] TS c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097): tablet 6c9adf5004a649979b9abb97df5c0c28 (table rebalance_test_table_14 [id=a699ad9359e6422b8223c2844e565c67]) successfully deleted
I20240502 20:46:09.347167 7937 raft_consensus.cc:686] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEADER]: Becoming Leader. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Running, Role: LEADER
I20240502 20:46:09.347285 7937 consensus_queue.cc:237] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:09.348640 7297 ts_tablet_manager.cc:1888] T 04c72d7d63574dea99cb2f7c14199a73 P c92e6e46782849479c8a0c6d82276fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:09.348701 7297 log.cc:1196] T 04c72d7d63574dea99cb2f7c14199a73 P c92e6e46782849479c8a0c6d82276fb2: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal/wals/04c72d7d63574dea99cb2f7c14199a73
I20240502 20:46:09.349118 6831 raft_consensus.cc:1203] T ed6386b6777c4d50b29ff8cd49894eba P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:09.349864 7297 ts_tablet_manager.cc:1909] T 04c72d7d63574dea99cb2f7c14199a73 P c92e6e46782849479c8a0c6d82276fb2: Deleting consensus metadata
I20240502 20:46:09.350463 4228 catalog_manager.cc:5484] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65). New cstate: current_term: 1 leader_uuid: "c92e6e46782849479c8a0c6d82276fb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:09.350566 4222 catalog_manager.cc:4830] TS c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827): tablet 04c72d7d63574dea99cb2f7c14199a73 (table rebalance_test_table_14 [id=a699ad9359e6422b8223c2844e565c67]) successfully deleted
I20240502 20:46:09.351229 7299 tablet_replica.cc:330] T 04c72d7d63574dea99cb2f7c14199a73 P ed47c023452a497192f35d2f617bf9bd: stopping tablet replica
I20240502 20:46:09.351387 7299 raft_consensus.cc:2229] T 04c72d7d63574dea99cb2f7c14199a73 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:09.351437 4222 catalog_manager.cc:4830] TS ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193): tablet e5a4dc24914c4f37a14481b77c4fcdaa (table rebalance_test_table_14 [id=a699ad9359e6422b8223c2844e565c67]) successfully deleted
I20240502 20:46:09.351541 7299 raft_consensus.cc:2258] T 04c72d7d63574dea99cb2f7c14199a73 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 20:46:09.351753 4219 catalog_manager.cc:4810] TS 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): delete failed for tablet e5a4dc24914c4f37a14481b77c4fcdaa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e5a4dc24914c4f37a14481b77c4fcdaa already in progress: copying tablet
I20240502 20:46:09.352190 7299 ts_tablet_manager.cc:1875] T 04c72d7d63574dea99cb2f7c14199a73 P ed47c023452a497192f35d2f617bf9bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:09.355079 7299 ts_tablet_manager.cc:1888] T 04c72d7d63574dea99cb2f7c14199a73 P ed47c023452a497192f35d2f617bf9bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:09.355222 7299 log.cc:1196] T 04c72d7d63574dea99cb2f7c14199a73 P ed47c023452a497192f35d2f617bf9bd: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal/wals/04c72d7d63574dea99cb2f7c14199a73
I20240502 20:46:09.355537 7299 ts_tablet_manager.cc:1909] T 04c72d7d63574dea99cb2f7c14199a73 P ed47c023452a497192f35d2f617bf9bd: Deleting consensus metadata
I20240502 20:46:09.356025 8676 tablet_bootstrap.cc:492] T e5a4dc24914c4f37a14481b77c4fcdaa P 201d2b6485084e7dad2cbb722af94779: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:09.356313 7299 tablet_replica.cc:330] T 6c9adf5004a649979b9abb97df5c0c28 P ed47c023452a497192f35d2f617bf9bd: stopping tablet replica
I20240502 20:46:09.356324 8676 tablet_bootstrap.cc:492] T e5a4dc24914c4f37a14481b77c4fcdaa P 201d2b6485084e7dad2cbb722af94779: Bootstrap complete.
I20240502 20:46:09.356353 4222 catalog_manager.cc:4830] TS ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193): tablet 04c72d7d63574dea99cb2f7c14199a73 (table rebalance_test_table_14 [id=a699ad9359e6422b8223c2844e565c67]) successfully deleted
I20240502 20:46:09.356557 8676 ts_tablet_manager.cc:1367] T e5a4dc24914c4f37a14481b77c4fcdaa P 201d2b6485084e7dad2cbb722af94779: Time spent bootstrapping tablet: real 0.051s user 0.029s sys 0.000s
I20240502 20:46:09.356734 6835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b7f7de1f8c49389c3d8c1e03c67f26" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833"
I20240502 20:46:09.356698 6831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b7f7de1f8c49389c3d8c1e03c67f26" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833" is_pre_election: true
I20240502 20:46:09.356822 6835 raft_consensus.cc:3037] T 93b7f7de1f8c49389c3d8c1e03c67f26 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:09.357082 8676 raft_consensus.cc:348] T e5a4dc24914c4f37a14481b77c4fcdaa P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } }
I20240502 20:46:09.357152 7299 raft_consensus.cc:2229] T 6c9adf5004a649979b9abb97df5c0c28 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:09.357210 8676 raft_consensus.cc:729] T e5a4dc24914c4f37a14481b77c4fcdaa P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Initialized, Role: LEARNER
I20240502 20:46:09.357306 8676 consensus_queue.cc:260] T e5a4dc24914c4f37a14481b77c4fcdaa P 201d2b6485084e7dad2cbb722af94779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } }
I20240502 20:46:09.357365 7299 raft_consensus.cc:2258] T 6c9adf5004a649979b9abb97df5c0c28 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:09.357602 8676 ts_tablet_manager.cc:1398] T e5a4dc24914c4f37a14481b77c4fcdaa P 201d2b6485084e7dad2cbb722af94779: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:09.357874 6835 raft_consensus.cc:2454] T 93b7f7de1f8c49389c3d8c1e03c67f26 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 1.
I20240502 20:46:09.358032 7299 ts_tablet_manager.cc:1875] T 6c9adf5004a649979b9abb97df5c0c28 P ed47c023452a497192f35d2f617bf9bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:09.358044 4790 tablet_copy_service.cc:342] P ed47c023452a497192f35d2f617bf9bd: Request end of tablet copy session 201d2b6485084e7dad2cbb722af94779-e5a4dc24914c4f37a14481b77c4fcdaa received from {username='slave'} at 127.30.32.72:39947
I20240502 20:46:09.358150 4790 tablet_copy_service.cc:433] P ed47c023452a497192f35d2f617bf9bd: ending tablet copy session 201d2b6485084e7dad2cbb722af94779-e5a4dc24914c4f37a14481b77c4fcdaa on tablet e5a4dc24914c4f37a14481b77c4fcdaa with peer 201d2b6485084e7dad2cbb722af94779
I20240502 20:46:09.359218 7079 tablet_service.cc:1514] Processing DeleteTablet for tablet e5a4dc24914c4f37a14481b77c4fcdaa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:09 UTC) from {username='slave'} at 127.0.0.1:59162
I20240502 20:46:09.359396 8805 tablet_replica.cc:330] T e5a4dc24914c4f37a14481b77c4fcdaa P 201d2b6485084e7dad2cbb722af94779: stopping tablet replica
I20240502 20:46:09.359537 8805 raft_consensus.cc:2229] T e5a4dc24914c4f37a14481b77c4fcdaa P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Raft consensus shutting down.
I20240502 20:46:09.359639 8805 raft_consensus.cc:2258] T e5a4dc24914c4f37a14481b77c4fcdaa P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Raft consensus is shut down!
I20240502 20:46:09.359954 7299 ts_tablet_manager.cc:1888] T 6c9adf5004a649979b9abb97df5c0c28 P ed47c023452a497192f35d2f617bf9bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 20:46:09.360019 7299 log.cc:1196] T 6c9adf5004a649979b9abb97df5c0c28 P ed47c023452a497192f35d2f617bf9bd: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal/wals/6c9adf5004a649979b9abb97df5c0c28
I20240502 20:46:09.360163 8805 ts_tablet_manager.cc:1875] T e5a4dc24914c4f37a14481b77c4fcdaa P 201d2b6485084e7dad2cbb722af94779: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:09.360292 7299 ts_tablet_manager.cc:1909] T 6c9adf5004a649979b9abb97df5c0c28 P ed47c023452a497192f35d2f617bf9bd: Deleting consensus metadata
I20240502 20:46:09.360845 4222 catalog_manager.cc:4830] TS ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193): tablet 6c9adf5004a649979b9abb97df5c0c28 (table rebalance_test_table_14 [id=a699ad9359e6422b8223c2844e565c67]) successfully deleted
I20240502 20:46:09.362119 8805 ts_tablet_manager.cc:1888] T e5a4dc24914c4f37a14481b77c4fcdaa P 201d2b6485084e7dad2cbb722af94779: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:09.362200 8805 log.cc:1196] T e5a4dc24914c4f37a14481b77c4fcdaa P 201d2b6485084e7dad2cbb722af94779: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/wal/wals/e5a4dc24914c4f37a14481b77c4fcdaa
I20240502 20:46:09.362514 8805 ts_tablet_manager.cc:1909] T e5a4dc24914c4f37a14481b77c4fcdaa P 201d2b6485084e7dad2cbb722af94779: Deleting consensus metadata
I20240502 20:46:09.363109 4219 catalog_manager.cc:4830] TS 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): tablet e5a4dc24914c4f37a14481b77c4fcdaa (table rebalance_test_table_14 [id=a699ad9359e6422b8223c2844e565c67]) successfully deleted
I20240502 20:46:09.388646 4228 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 20:46:09.394336 8776 tablet.cc:1722] T a88bdfa2bca8401d99eec723457a1125 P 7845c397230d49358ba2fad9db53f97c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 20:46:09.394701 8776 tablet.cc:1620] T a88bdfa2bca8401d99eec723457a1125 P 7845c397230d49358ba2fad9db53f97c: MemRowSet was empty: no flush needed.
I20240502 20:46:09.394887 8778 tablet.cc:1722] T a88bdfa2bca8401d99eec723457a1125 P 923cfab926e7443f8ffd7b21ad0a9d91: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 20:46:09.395004 8777 tablet.cc:1722] T a88bdfa2bca8401d99eec723457a1125 P a4bf1df7fa6b4948be502c182bfbb833: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 20:46:09.395159 8778 tablet.cc:1620] T a88bdfa2bca8401d99eec723457a1125 P 923cfab926e7443f8ffd7b21ad0a9d91: MemRowSet was empty: no flush needed.
I20240502 20:46:09.395274 8777 tablet.cc:1620] T a88bdfa2bca8401d99eec723457a1125 P a4bf1df7fa6b4948be502c182bfbb833: MemRowSet was empty: no flush needed.
I20240502 20:46:09.400732 4230 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: rebalancer_dynamic_table [id=9a174cd46c574a628620ba4cba9d6466] alter complete (version 2)
I20240502 20:46:09.416823 6967 raft_consensus.cc:1203] T 0cf7e95cb91b4e74b14365fcc6b7f968 P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:09.421116 8676 ts_tablet_manager.cc:902] T eebddad3efe748f78372901ecad5b407 P 201d2b6485084e7dad2cbb722af94779: Initiating tablet copy from peer 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:46:09.421651 8676 tablet_copy_client.cc:308] T eebddad3efe748f78372901ecad5b407 P 201d2b6485084e7dad2cbb722af94779: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.67:42141
I20240502 20:46:09.425547 4658 tablet_copy_service.cc:140] P 923cfab926e7443f8ffd7b21ad0a9d91: Received BeginTabletCopySession request for tablet eebddad3efe748f78372901ecad5b407 from peer 201d2b6485084e7dad2cbb722af94779 ({username='slave'} at 127.30.32.72:36609)
I20240502 20:46:09.425684 4658 tablet_copy_service.cc:161] P 923cfab926e7443f8ffd7b21ad0a9d91: Beginning new tablet copy session on tablet eebddad3efe748f78372901ecad5b407 from peer 201d2b6485084e7dad2cbb722af94779 at {username='slave'} at 127.30.32.72:36609: session id = 201d2b6485084e7dad2cbb722af94779-eebddad3efe748f78372901ecad5b407
I20240502 20:46:09.426772 4658 tablet_copy_source_session.cc:205] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:09.427729 8676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eebddad3efe748f78372901ecad5b407. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:09.429839 8676 tablet_copy_client.cc:783] T eebddad3efe748f78372901ecad5b407 P 201d2b6485084e7dad2cbb722af94779: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:09.430008 8676 tablet_copy_client.cc:647] T eebddad3efe748f78372901ecad5b407 P 201d2b6485084e7dad2cbb722af94779: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:09.431284 8676 tablet_copy_client.cc:523] T eebddad3efe748f78372901ecad5b407 P 201d2b6485084e7dad2cbb722af94779: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:09.431316 8814 ts_tablet_manager.cc:902] T c55ba788f592448d8941ff94a8cefc52 P a47e5c56f13843018ac584914a8d4707: Initiating tablet copy from peer c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:46:09.431828 8814 tablet_copy_client.cc:308] T c55ba788f592448d8941ff94a8cefc52 P a47e5c56f13843018ac584914a8d4707: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.65:33827
I20240502 20:46:09.432332 4397 tablet_copy_service.cc:140] P c92e6e46782849479c8a0c6d82276fb2: Received BeginTabletCopySession request for tablet c55ba788f592448d8941ff94a8cefc52 from peer a47e5c56f13843018ac584914a8d4707 ({username='slave'} at 127.30.32.71:33061)
I20240502 20:46:09.432443 4397 tablet_copy_service.cc:161] P c92e6e46782849479c8a0c6d82276fb2: Beginning new tablet copy session on tablet c55ba788f592448d8941ff94a8cefc52 from peer a47e5c56f13843018ac584914a8d4707 at {username='slave'} at 127.30.32.71:33061: session id = a47e5c56f13843018ac584914a8d4707-c55ba788f592448d8941ff94a8cefc52
I20240502 20:46:09.432616 8676 tablet_bootstrap.cc:492] T eebddad3efe748f78372901ecad5b407 P 201d2b6485084e7dad2cbb722af94779: Bootstrap starting.
I20240502 20:46:09.433555 4397 tablet_copy_source_session.cc:205] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:09.434440 8814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c55ba788f592448d8941ff94a8cefc52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:09.436811 8814 tablet_copy_client.cc:783] T c55ba788f592448d8941ff94a8cefc52 P a47e5c56f13843018ac584914a8d4707: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:09.437134 8814 tablet_copy_client.cc:647] T c55ba788f592448d8941ff94a8cefc52 P a47e5c56f13843018ac584914a8d4707: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:09.438410 8814 tablet_copy_client.cc:523] T c55ba788f592448d8941ff94a8cefc52 P a47e5c56f13843018ac584914a8d4707: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:09.439388 8814 tablet_bootstrap.cc:492] T c55ba788f592448d8941ff94a8cefc52 P a47e5c56f13843018ac584914a8d4707: Bootstrap starting.
I20240502 20:46:09.454222 8676 tablet_bootstrap.cc:492] T eebddad3efe748f78372901ecad5b407 P 201d2b6485084e7dad2cbb722af94779: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:09.454530 8676 tablet_bootstrap.cc:492] T eebddad3efe748f78372901ecad5b407 P 201d2b6485084e7dad2cbb722af94779: Bootstrap complete.
I20240502 20:46:09.454731 8676 ts_tablet_manager.cc:1367] T eebddad3efe748f78372901ecad5b407 P 201d2b6485084e7dad2cbb722af94779: Time spent bootstrapping tablet: real 0.022s user 0.017s sys 0.003s
I20240502 20:46:09.455185 8676 raft_consensus.cc:348] T eebddad3efe748f78372901ecad5b407 P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } }
I20240502 20:46:09.455308 8676 raft_consensus.cc:729] T eebddad3efe748f78372901ecad5b407 P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Initialized, Role: LEARNER
I20240502 20:46:09.455425 8676 consensus_queue.cc:260] T eebddad3efe748f78372901ecad5b407 P 201d2b6485084e7dad2cbb722af94779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } }
I20240502 20:46:09.455724 8676 ts_tablet_manager.cc:1398] T eebddad3efe748f78372901ecad5b407 P 201d2b6485084e7dad2cbb722af94779: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:09.457361 4658 tablet_copy_service.cc:342] P 923cfab926e7443f8ffd7b21ad0a9d91: Request end of tablet copy session 201d2b6485084e7dad2cbb722af94779-eebddad3efe748f78372901ecad5b407 received from {username='slave'} at 127.30.32.72:36609
I20240502 20:46:09.457453 4658 tablet_copy_service.cc:433] P 923cfab926e7443f8ffd7b21ad0a9d91: ending tablet copy session 201d2b6485084e7dad2cbb722af94779-eebddad3efe748f78372901ecad5b407 on tablet eebddad3efe748f78372901ecad5b407 with peer 201d2b6485084e7dad2cbb722af94779
I20240502 20:46:09.461259 8814 tablet_bootstrap.cc:492] T c55ba788f592448d8941ff94a8cefc52 P a47e5c56f13843018ac584914a8d4707: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:09.461517 8814 tablet_bootstrap.cc:492] T c55ba788f592448d8941ff94a8cefc52 P a47e5c56f13843018ac584914a8d4707: Bootstrap complete.
I20240502 20:46:09.461664 8814 ts_tablet_manager.cc:1367] T c55ba788f592448d8941ff94a8cefc52 P a47e5c56f13843018ac584914a8d4707: Time spent bootstrapping tablet: real 0.022s user 0.022s sys 0.000s
I20240502 20:46:09.462023 8814 raft_consensus.cc:348] T c55ba788f592448d8941ff94a8cefc52 P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } }
I20240502 20:46:09.462126 8814 raft_consensus.cc:729] T c55ba788f592448d8941ff94a8cefc52 P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Initialized, Role: LEARNER
I20240502 20:46:09.462206 8814 consensus_queue.cc:260] T c55ba788f592448d8941ff94a8cefc52 P a47e5c56f13843018ac584914a8d4707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } }
I20240502 20:46:09.462443 8814 ts_tablet_manager.cc:1398] T c55ba788f592448d8941ff94a8cefc52 P a47e5c56f13843018ac584914a8d4707: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:09.462970 4397 tablet_copy_service.cc:342] P c92e6e46782849479c8a0c6d82276fb2: Request end of tablet copy session a47e5c56f13843018ac584914a8d4707-c55ba788f592448d8941ff94a8cefc52 received from {username='slave'} at 127.30.32.71:33061
I20240502 20:46:09.463083 4397 tablet_copy_service.cc:433] P c92e6e46782849479c8a0c6d82276fb2: ending tablet copy session a47e5c56f13843018ac584914a8d4707-c55ba788f592448d8941ff94a8cefc52 on tablet c55ba788f592448d8941ff94a8cefc52 with peer a47e5c56f13843018ac584914a8d4707
I20240502 20:46:09.502952 4233 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 20:46:09.508626 8776 tablet.cc:1722] T a88bdfa2bca8401d99eec723457a1125 P 7845c397230d49358ba2fad9db53f97c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 20:46:09.509311 8777 tablet.cc:1722] T a88bdfa2bca8401d99eec723457a1125 P a4bf1df7fa6b4948be502c182bfbb833: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 20:46:09.509579 8778 tablet.cc:1722] T a88bdfa2bca8401d99eec723457a1125 P 923cfab926e7443f8ffd7b21ad0a9d91: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 20:46:09.510581 7099 raft_consensus.cc:1203] T cdf218d5e45b4db19a8e2c22dbc5cc5d P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:09.514681 4228 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: rebalancer_dynamic_table_new_name [id=9a174cd46c574a628620ba4cba9d6466] alter complete (version 3)
I20240502 20:46:09.529587 8814 ts_tablet_manager.cc:902] T 7abfd61f55554042978b50761cbd6ab5 P a47e5c56f13843018ac584914a8d4707: Initiating tablet copy from peer 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:46:09.530007 8814 tablet_copy_client.cc:308] T 7abfd61f55554042978b50761cbd6ab5 P a47e5c56f13843018ac584914a8d4707: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.67:42141
I20240502 20:46:09.530455 4658 tablet_copy_service.cc:140] P 923cfab926e7443f8ffd7b21ad0a9d91: Received BeginTabletCopySession request for tablet 7abfd61f55554042978b50761cbd6ab5 from peer a47e5c56f13843018ac584914a8d4707 ({username='slave'} at 127.30.32.71:59213)
I20240502 20:46:09.530550 4658 tablet_copy_service.cc:161] P 923cfab926e7443f8ffd7b21ad0a9d91: Beginning new tablet copy session on tablet 7abfd61f55554042978b50761cbd6ab5 from peer a47e5c56f13843018ac584914a8d4707 at {username='slave'} at 127.30.32.71:59213: session id = a47e5c56f13843018ac584914a8d4707-7abfd61f55554042978b50761cbd6ab5
I20240502 20:46:09.531560 4658 tablet_copy_source_session.cc:205] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:09.532912 8814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7abfd61f55554042978b50761cbd6ab5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:09.533105 6835 raft_consensus.cc:1203] T 5053a7afb4994ecdbc92b129c58903cf P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:09.535310 8814 tablet_copy_client.cc:783] T 7abfd61f55554042978b50761cbd6ab5 P a47e5c56f13843018ac584914a8d4707: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:09.535465 8814 tablet_copy_client.cc:647] T 7abfd61f55554042978b50761cbd6ab5 P a47e5c56f13843018ac584914a8d4707: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:09.537003 8814 tablet_copy_client.cc:523] T 7abfd61f55554042978b50761cbd6ab5 P a47e5c56f13843018ac584914a8d4707: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:09.538312 8814 tablet_bootstrap.cc:492] T 7abfd61f55554042978b50761cbd6ab5 P a47e5c56f13843018ac584914a8d4707: Bootstrap starting.
I20240502 20:46:09.548491 8189 consensus_queue.cc:1035] T 52e302882311488ab1cac90ad03fe270 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:09.554108 8364 consensus_queue.cc:1035] T 52e302882311488ab1cac90ad03fe270 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:09.570024 8814 tablet_bootstrap.cc:492] T 7abfd61f55554042978b50761cbd6ab5 P a47e5c56f13843018ac584914a8d4707: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:09.570338 8814 tablet_bootstrap.cc:492] T 7abfd61f55554042978b50761cbd6ab5 P a47e5c56f13843018ac584914a8d4707: Bootstrap complete.
I20240502 20:46:09.570514 8814 ts_tablet_manager.cc:1367] T 7abfd61f55554042978b50761cbd6ab5 P a47e5c56f13843018ac584914a8d4707: Time spent bootstrapping tablet: real 0.032s user 0.026s sys 0.000s
I20240502 20:46:09.570938 8814 raft_consensus.cc:348] T 7abfd61f55554042978b50761cbd6ab5 P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } }
I20240502 20:46:09.571046 8814 raft_consensus.cc:729] T 7abfd61f55554042978b50761cbd6ab5 P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Initialized, Role: LEARNER
I20240502 20:46:09.571146 8814 consensus_queue.cc:260] T 7abfd61f55554042978b50761cbd6ab5 P a47e5c56f13843018ac584914a8d4707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } }
I20240502 20:46:09.571444 8814 ts_tablet_manager.cc:1398] T 7abfd61f55554042978b50761cbd6ab5 P a47e5c56f13843018ac584914a8d4707: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:09.573113 4658 tablet_copy_service.cc:342] P 923cfab926e7443f8ffd7b21ad0a9d91: Request end of tablet copy session a47e5c56f13843018ac584914a8d4707-7abfd61f55554042978b50761cbd6ab5 received from {username='slave'} at 127.30.32.71:59213
I20240502 20:46:09.573207 4658 tablet_copy_service.cc:433] P 923cfab926e7443f8ffd7b21ad0a9d91: ending tablet copy session a47e5c56f13843018ac584914a8d4707-7abfd61f55554042978b50761cbd6ab5 on tablet 7abfd61f55554042978b50761cbd6ab5 with peer a47e5c56f13843018ac584914a8d4707
I20240502 20:46:09.584097 4374 consensus_queue.cc:237] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } }
I20240502 20:46:09.585449 7099 raft_consensus.cc:1261] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 201d2b6485084e7dad2cbb722af94779 [term 2 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 20:46:09.585441 4638 raft_consensus.cc:1261] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 2 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 20:46:09.585980 8185 consensus_queue.cc:1035] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:09.586557 8185 consensus_queue.cc:1035] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:09.589300 7937 raft_consensus.cc:2932] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c92e6e46782849479c8a0c6d82276fb2 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:09.589924 7099 raft_consensus.cc:2932] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 201d2b6485084e7dad2cbb722af94779 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:09.591282 4222 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5a36bf6e8e6e43b2b47b6d6357b0d445 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:09.592643 4638 raft_consensus.cc:2932] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:09.593199 4232 catalog_manager.cc:5484] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c92e6e46782849479c8a0c6d82276fb2 reported cstate change: config changed from index 5 to 7, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New cstate: current_term: 2 leader_uuid: "c92e6e46782849479c8a0c6d82276fb2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:09.596707 4487 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a36bf6e8e6e43b2b47b6d6357b0d445 with delete_type TABLET_DATA_TOMBSTONED (TS c3d26eafd51547acb780d4aac5ebe7cc not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:48890
I20240502 20:46:09.597309 8141 tablet_replica.cc:330] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc: stopping tablet replica
I20240502 20:46:09.597486 8141 raft_consensus.cc:2229] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:09.597604 8141 raft_consensus.cc:2258] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:09.598433 8141 ts_tablet_manager.cc:1875] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:09.600936 8141 ts_tablet_manager.cc:1888] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 20:46:09.601028 8141 log.cc:1196] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal/wals/5a36bf6e8e6e43b2b47b6d6357b0d445
I20240502 20:46:09.601593 4221 catalog_manager.cc:4830] TS c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097): tablet 5a36bf6e8e6e43b2b47b6d6357b0d445 (table rebalance_test_table_16 [id=dffd2e92869942b9a6f03505945ad3ba]) successfully deleted
I20240502 20:46:09.618443 4233 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 20:46:09.618598 4233 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 20:46:09.620743 4233 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: Sending DeleteTablet for 3 replicas of tablet a88bdfa2bca8401d99eec723457a1125
I20240502 20:46:09.620924 6702 raft_consensus.cc:1203] T a55d5b8b9ed04ca39888b89b82a3f833 P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:09.621897 6682 tablet_service.cc:1514] Processing DeleteTablet for tablet a88bdfa2bca8401d99eec723457a1125 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:09 UTC) from {username='slave'} at 127.0.0.1:37506
I20240502 20:46:09.622049 6815 tablet_service.cc:1514] Processing DeleteTablet for tablet a88bdfa2bca8401d99eec723457a1125 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:09 UTC) from {username='slave'} at 127.0.0.1:35290
I20240502 20:46:09.622093 4618 tablet_service.cc:1514] Processing DeleteTablet for tablet a88bdfa2bca8401d99eec723457a1125 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:09 UTC) from {username='slave'} at 127.0.0.1:57068
I20240502 20:46:09.622287 8806 tablet_replica.cc:330] T a88bdfa2bca8401d99eec723457a1125 P 7845c397230d49358ba2fad9db53f97c: stopping tablet replica
I20240502 20:46:09.622299 8510 tablet_replica.cc:330] T a88bdfa2bca8401d99eec723457a1125 P 923cfab926e7443f8ffd7b21ad0a9d91: stopping tablet replica
I20240502 20:46:09.622393 8806 raft_consensus.cc:2229] T a88bdfa2bca8401d99eec723457a1125 P 7845c397230d49358ba2fad9db53f97c [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:09.622484 8510 raft_consensus.cc:2229] T a88bdfa2bca8401d99eec723457a1125 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:09.622627 8510 raft_consensus.cc:2258] T a88bdfa2bca8401d99eec723457a1125 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:09.622646 8806 raft_consensus.cc:2258] T a88bdfa2bca8401d99eec723457a1125 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:09.623356 8806 ts_tablet_manager.cc:1875] T a88bdfa2bca8401d99eec723457a1125 P 7845c397230d49358ba2fad9db53f97c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:09.624075 8510 ts_tablet_manager.cc:1875] T a88bdfa2bca8401d99eec723457a1125 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:09.624226 8900 tablet_replica.cc:330] T a88bdfa2bca8401d99eec723457a1125 P a4bf1df7fa6b4948be502c182bfbb833: stopping tablet replica
I20240502 20:46:09.624368 8900 raft_consensus.cc:2229] T a88bdfa2bca8401d99eec723457a1125 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:09.624505 8900 raft_consensus.cc:2258] T a88bdfa2bca8401d99eec723457a1125 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:09.625614 8806 ts_tablet_manager.cc:1888] T a88bdfa2bca8401d99eec723457a1125 P 7845c397230d49358ba2fad9db53f97c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:09.625696 8806 log.cc:1196] T a88bdfa2bca8401d99eec723457a1125 P 7845c397230d49358ba2fad9db53f97c: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/wal/wals/a88bdfa2bca8401d99eec723457a1125
I20240502 20:46:09.625738 8900 ts_tablet_manager.cc:1875] T a88bdfa2bca8401d99eec723457a1125 P a4bf1df7fa6b4948be502c182bfbb833: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:09.626009 8806 ts_tablet_manager.cc:1909] T a88bdfa2bca8401d99eec723457a1125 P 7845c397230d49358ba2fad9db53f97c: Deleting consensus metadata
I20240502 20:46:09.626641 8510 ts_tablet_manager.cc:1888] T a88bdfa2bca8401d99eec723457a1125 P 923cfab926e7443f8ffd7b21ad0a9d91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:09.626833 4221 catalog_manager.cc:4830] TS 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): tablet a88bdfa2bca8401d99eec723457a1125 (table rebalancer_dynamic_table_new_name [id=9a174cd46c574a628620ba4cba9d6466]) successfully deleted
I20240502 20:46:09.627167 8510 log.cc:1196] T a88bdfa2bca8401d99eec723457a1125 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal/wals/a88bdfa2bca8401d99eec723457a1125
I20240502 20:46:09.627486 8510 ts_tablet_manager.cc:1909] T a88bdfa2bca8401d99eec723457a1125 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting consensus metadata
I20240502 20:46:09.627938 8900 ts_tablet_manager.cc:1888] T a88bdfa2bca8401d99eec723457a1125 P a4bf1df7fa6b4948be502c182bfbb833: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:09.628070 8900 log.cc:1196] T a88bdfa2bca8401d99eec723457a1125 P a4bf1df7fa6b4948be502c182bfbb833: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/wal/wals/a88bdfa2bca8401d99eec723457a1125
I20240502 20:46:09.628180 4219 catalog_manager.cc:4830] TS 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141): tablet a88bdfa2bca8401d99eec723457a1125 (table rebalancer_dynamic_table_new_name [id=9a174cd46c574a628620ba4cba9d6466]) successfully deleted
I20240502 20:46:09.628351 8900 ts_tablet_manager.cc:1909] T a88bdfa2bca8401d99eec723457a1125 P a4bf1df7fa6b4948be502c182bfbb833: Deleting consensus metadata
I20240502 20:46:09.629021 4220 catalog_manager.cc:4830] TS a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867): tablet a88bdfa2bca8401d99eec723457a1125 (table rebalancer_dynamic_table_new_name [id=9a174cd46c574a628620ba4cba9d6466]) successfully deleted
I20240502 20:46:09.638957 8146 raft_consensus.cc:1050] T 66b8f583dfd24c2aa5a59457a8e58fbb P c3d26eafd51547acb780d4aac5ebe7cc: attempting to promote NON_VOTER a47e5c56f13843018ac584914a8d4707 to VOTER
I20240502 20:46:09.639428 8146 consensus_queue.cc:237] T 66b8f583dfd24c2aa5a59457a8e58fbb P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } }
I20240502 20:46:09.641168 6967 raft_consensus.cc:1261] T 66b8f583dfd24c2aa5a59457a8e58fbb P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:09.641212 4374 raft_consensus.cc:1261] T 66b8f583dfd24c2aa5a59457a8e58fbb P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:09.641873 7967 consensus_queue.cc:1035] T 66b8f583dfd24c2aa5a59457a8e58fbb P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:09.641947 4769 raft_consensus.cc:1261] T 66b8f583dfd24c2aa5a59457a8e58fbb P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:09.642252 7967 consensus_queue.cc:1035] T 66b8f583dfd24c2aa5a59457a8e58fbb P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:09.642585 7967 consensus_queue.cc:1035] T 66b8f583dfd24c2aa5a59457a8e58fbb P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:09.644521 6375 raft_consensus.cc:2932] T 66b8f583dfd24c2aa5a59457a8e58fbb P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:09.645198 4769 raft_consensus.cc:2932] T 66b8f583dfd24c2aa5a59457a8e58fbb P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:09.645447 4374 raft_consensus.cc:2932] T 66b8f583dfd24c2aa5a59457a8e58fbb P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:09.647589 6967 raft_consensus.cc:2932] T 66b8f583dfd24c2aa5a59457a8e58fbb P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:09.648344 4227 catalog_manager.cc:5484] T 66b8f583dfd24c2aa5a59457a8e58fbb P ed47c023452a497192f35d2f617bf9bd reported cstate change: config changed from index 4 to 5, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:09.652318 4506 consensus_queue.cc:237] T 66b8f583dfd24c2aa5a59457a8e58fbb P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } }
I20240502 20:46:09.653559 6967 raft_consensus.cc:1261] T 66b8f583dfd24c2aa5a59457a8e58fbb P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:09.653658 4374 raft_consensus.cc:1261] T 66b8f583dfd24c2aa5a59457a8e58fbb P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:09.653964 7967 consensus_queue.cc:1035] T 66b8f583dfd24c2aa5a59457a8e58fbb P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:09.654124 8907 consensus_queue.cc:1035] T 66b8f583dfd24c2aa5a59457a8e58fbb P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:09.655923 8146 raft_consensus.cc:2932] T 66b8f583dfd24c2aa5a59457a8e58fbb P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:09.656242 4374 raft_consensus.cc:2932] T 66b8f583dfd24c2aa5a59457a8e58fbb P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:09.657331 4221 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 66b8f583dfd24c2aa5a59457a8e58fbb with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:09.657256 6967 raft_consensus.cc:2932] T 66b8f583dfd24c2aa5a59457a8e58fbb P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:09.659536 4233 catalog_manager.cc:5484] T 66b8f583dfd24c2aa5a59457a8e58fbb P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: config changed from index 5 to 6, VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) evicted. New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:09.661940 4750 tablet_service.cc:1514] Processing DeleteTablet for tablet 66b8f583dfd24c2aa5a59457a8e58fbb with delete_type TABLET_DATA_TOMBSTONED (TS ed47c023452a497192f35d2f617bf9bd not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38526
I20240502 20:46:09.662186 7299 tablet_replica.cc:330] T 66b8f583dfd24c2aa5a59457a8e58fbb P ed47c023452a497192f35d2f617bf9bd: stopping tablet replica
I20240502 20:46:09.662331 7299 raft_consensus.cc:2229] T 66b8f583dfd24c2aa5a59457a8e58fbb P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:09.662492 7299 raft_consensus.cc:2258] T 66b8f583dfd24c2aa5a59457a8e58fbb P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:09.663177 7299 ts_tablet_manager.cc:1875] T 66b8f583dfd24c2aa5a59457a8e58fbb P ed47c023452a497192f35d2f617bf9bd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:09.664103 4374 consensus_queue.cc:237] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } }
I20240502 20:46:09.665303 8556 ts_tablet_manager.cc:902] T dbc1ef85e30d438682bc86935ceefd1b P a4bf1df7fa6b4948be502c182bfbb833: Initiating tablet copy from peer 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:46:09.665617 7299 ts_tablet_manager.cc:1888] T 66b8f583dfd24c2aa5a59457a8e58fbb P ed47c023452a497192f35d2f617bf9bd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:46:09.665692 7299 log.cc:1196] T 66b8f583dfd24c2aa5a59457a8e58fbb P ed47c023452a497192f35d2f617bf9bd: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal/wals/66b8f583dfd24c2aa5a59457a8e58fbb
I20240502 20:46:09.665850 8556 tablet_copy_client.cc:308] T dbc1ef85e30d438682bc86935ceefd1b P a4bf1df7fa6b4948be502c182bfbb833: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.67:42141
I20240502 20:46:09.666508 4222 catalog_manager.cc:4830] TS ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193): tablet 66b8f583dfd24c2aa5a59457a8e58fbb (table rebalance_test_table_18 [id=393c8bbf6b174a9e891d96bcd09fe045]) successfully deleted
I20240502 20:46:09.666535 4506 raft_consensus.cc:1261] T 15a2cbe8285a4938be7f628f7f8b734d P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:09.666635 7099 raft_consensus.cc:1261] T 15a2cbe8285a4938be7f628f7f8b734d P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:09.666890 4658 tablet_copy_service.cc:140] P 923cfab926e7443f8ffd7b21ad0a9d91: Received BeginTabletCopySession request for tablet dbc1ef85e30d438682bc86935ceefd1b from peer a4bf1df7fa6b4948be502c182bfbb833 ({username='slave'} at 127.30.32.70:32901)
I20240502 20:46:09.666997 4658 tablet_copy_service.cc:161] P 923cfab926e7443f8ffd7b21ad0a9d91: Beginning new tablet copy session on tablet dbc1ef85e30d438682bc86935ceefd1b from peer a4bf1df7fa6b4948be502c182bfbb833 at {username='slave'} at 127.30.32.70:32901: session id = a4bf1df7fa6b4948be502c182bfbb833-dbc1ef85e30d438682bc86935ceefd1b
I20240502 20:46:09.667078 8918 consensus_queue.cc:1035] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:46:09.667959 4658 tablet_copy_source_session.cc:205] T dbc1ef85e30d438682bc86935ceefd1b P 923cfab926e7443f8ffd7b21ad0a9d91: Tablet Copy: opened 0 blocks and 1 log segments
W20240502 20:46:09.668450 4310 consensus_peers.cc:485] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2 -> Peer 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141): Couldn't send request to peer 923cfab926e7443f8ffd7b21ad0a9d91. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15a2cbe8285a4938be7f628f7f8b734d. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:09.668556 8918 consensus_queue.cc:1035] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:46:09.668973 8918 raft_consensus.cc:2932] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } } }
I20240502 20:46:09.669380 8556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbc1ef85e30d438682bc86935ceefd1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:09.669435 7099 raft_consensus.cc:2932] T 15a2cbe8285a4938be7f628f7f8b734d P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } } }
I20240502 20:46:09.671200 8556 tablet_copy_client.cc:783] T dbc1ef85e30d438682bc86935ceefd1b P a4bf1df7fa6b4948be502c182bfbb833: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:09.671339 8556 tablet_copy_client.cc:647] T dbc1ef85e30d438682bc86935ceefd1b P a4bf1df7fa6b4948be502c182bfbb833: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:09.672159 4232 catalog_manager.cc:5484] T 15a2cbe8285a4938be7f628f7f8b734d P 201d2b6485084e7dad2cbb722af94779 reported cstate change: config changed from index -1 to 2, NON_VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) added. New cstate: current_term: 1 leader_uuid: "c92e6e46782849479c8a0c6d82276fb2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } } }
I20240502 20:46:09.678236 8556 tablet_copy_client.cc:523] T dbc1ef85e30d438682bc86935ceefd1b P a4bf1df7fa6b4948be502c182bfbb833: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:09.679848 8556 tablet_bootstrap.cc:492] T dbc1ef85e30d438682bc86935ceefd1b P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap starting.
I20240502 20:46:09.685572 4506 consensus_queue.cc:237] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } }
I20240502 20:46:09.687171 6702 raft_consensus.cc:1261] T d6fb394fdcff436886fe0634d79c024c P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:09.687135 4506 raft_consensus.cc:2932] T 15a2cbe8285a4938be7f628f7f8b734d P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } } }
I20240502 20:46:09.687294 4769 raft_consensus.cc:1261] T d6fb394fdcff436886fe0634d79c024c P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:09.687722 8146 consensus_queue.cc:1035] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:09.688047 8146 consensus_queue.cc:1035] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 20:46:09.689090 4441 consensus_peers.cc:485] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d6fb394fdcff436886fe0634d79c024c. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:09.698020 8907 raft_consensus.cc:2932] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } } }
I20240502 20:46:09.698635 4769 raft_consensus.cc:2932] T d6fb394fdcff436886fe0634d79c024c P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } } }
I20240502 20:46:09.701987 4228 catalog_manager.cc:5484] T d6fb394fdcff436886fe0634d79c024c P ed47c023452a497192f35d2f617bf9bd reported cstate change: config changed from index 6 to 7, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } } }
I20240502 20:46:09.705765 6702 raft_consensus.cc:2932] T d6fb394fdcff436886fe0634d79c024c P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } } }
I20240502 20:46:09.709203 8556 tablet_bootstrap.cc:492] T dbc1ef85e30d438682bc86935ceefd1b P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:09.709641 8556 tablet_bootstrap.cc:492] T dbc1ef85e30d438682bc86935ceefd1b P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap complete.
I20240502 20:46:09.709828 8556 ts_tablet_manager.cc:1367] T dbc1ef85e30d438682bc86935ceefd1b P a4bf1df7fa6b4948be502c182bfbb833: Time spent bootstrapping tablet: real 0.030s user 0.021s sys 0.004s
I20240502 20:46:09.710346 8556 raft_consensus.cc:348] T dbc1ef85e30d438682bc86935ceefd1b P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } }
I20240502 20:46:09.710477 8556 raft_consensus.cc:729] T dbc1ef85e30d438682bc86935ceefd1b P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Initialized, Role: LEARNER
I20240502 20:46:09.710587 8556 consensus_queue.cc:260] T dbc1ef85e30d438682bc86935ceefd1b P a4bf1df7fa6b4948be502c182bfbb833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } }
I20240502 20:46:09.711402 8556 ts_tablet_manager.cc:1398] T dbc1ef85e30d438682bc86935ceefd1b P a4bf1df7fa6b4948be502c182bfbb833: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:09.711906 4658 tablet_copy_service.cc:342] P 923cfab926e7443f8ffd7b21ad0a9d91: Request end of tablet copy session a4bf1df7fa6b4948be502c182bfbb833-dbc1ef85e30d438682bc86935ceefd1b received from {username='slave'} at 127.30.32.70:32901
I20240502 20:46:09.711987 4658 tablet_copy_service.cc:433] P 923cfab926e7443f8ffd7b21ad0a9d91: ending tablet copy session a4bf1df7fa6b4948be502c182bfbb833-dbc1ef85e30d438682bc86935ceefd1b on tablet dbc1ef85e30d438682bc86935ceefd1b with peer a4bf1df7fa6b4948be502c182bfbb833
I20240502 20:46:09.726059 4228 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41812:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 20:46:09.731328 6947 tablet_service.cc:1467] Processing CreateTablet for tablet 1fc0c640a6fb4b24b17642ed26ce8f07 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=44b9d8eb907c483bb3ba9399264a26fb]), partition=
I20240502 20:46:09.731362 4750 tablet_service.cc:1467] Processing CreateTablet for tablet 1fc0c640a6fb4b24b17642ed26ce8f07 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=44b9d8eb907c483bb3ba9399264a26fb]), partition=
I20240502 20:46:09.731429 7079 tablet_service.cc:1467] Processing CreateTablet for tablet 1fc0c640a6fb4b24b17642ed26ce8f07 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=44b9d8eb907c483bb3ba9399264a26fb]), partition=
I20240502 20:46:09.731626 6947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fc0c640a6fb4b24b17642ed26ce8f07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:09.731635 4750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fc0c640a6fb4b24b17642ed26ce8f07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:09.731695 7079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fc0c640a6fb4b24b17642ed26ce8f07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:09.734532 8954 tablet_bootstrap.cc:492] T 1fc0c640a6fb4b24b17642ed26ce8f07 P a47e5c56f13843018ac584914a8d4707: Bootstrap starting.
I20240502 20:46:09.734668 8953 tablet_bootstrap.cc:492] T 1fc0c640a6fb4b24b17642ed26ce8f07 P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:46:09.735298 8955 tablet_bootstrap.cc:492] T 1fc0c640a6fb4b24b17642ed26ce8f07 P 201d2b6485084e7dad2cbb722af94779: Bootstrap starting.
I20240502 20:46:09.735641 8953 tablet_bootstrap.cc:654] T 1fc0c640a6fb4b24b17642ed26ce8f07 P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:09.735651 8954 tablet_bootstrap.cc:654] T 1fc0c640a6fb4b24b17642ed26ce8f07 P a47e5c56f13843018ac584914a8d4707: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:09.736912 8955 tablet_bootstrap.cc:654] T 1fc0c640a6fb4b24b17642ed26ce8f07 P 201d2b6485084e7dad2cbb722af94779: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:09.738328 8955 tablet_bootstrap.cc:492] T 1fc0c640a6fb4b24b17642ed26ce8f07 P 201d2b6485084e7dad2cbb722af94779: No bootstrap required, opened a new log
I20240502 20:46:09.738485 8954 tablet_bootstrap.cc:492] T 1fc0c640a6fb4b24b17642ed26ce8f07 P a47e5c56f13843018ac584914a8d4707: No bootstrap required, opened a new log
I20240502 20:46:09.738593 8954 ts_tablet_manager.cc:1367] T 1fc0c640a6fb4b24b17642ed26ce8f07 P a47e5c56f13843018ac584914a8d4707: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 20:46:09.738611 8955 ts_tablet_manager.cc:1367] T 1fc0c640a6fb4b24b17642ed26ce8f07 P 201d2b6485084e7dad2cbb722af94779: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240502 20:46:09.739054 8954 raft_consensus.cc:348] T 1fc0c640a6fb4b24b17642ed26ce8f07 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:09.739051 8955 raft_consensus.cc:348] T 1fc0c640a6fb4b24b17642ed26ce8f07 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:09.739166 8954 raft_consensus.cc:374] T 1fc0c640a6fb4b24b17642ed26ce8f07 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:09.739189 8954 raft_consensus.cc:729] T 1fc0c640a6fb4b24b17642ed26ce8f07 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Initialized, Role: FOLLOWER
I20240502 20:46:09.739179 8955 raft_consensus.cc:374] T 1fc0c640a6fb4b24b17642ed26ce8f07 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:09.739215 8955 raft_consensus.cc:729] T 1fc0c640a6fb4b24b17642ed26ce8f07 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Initialized, Role: FOLLOWER
I20240502 20:46:09.739312 8954 consensus_queue.cc:260] T 1fc0c640a6fb4b24b17642ed26ce8f07 P a47e5c56f13843018ac584914a8d4707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:09.739331 8955 consensus_queue.cc:260] T 1fc0c640a6fb4b24b17642ed26ce8f07 P 201d2b6485084e7dad2cbb722af94779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:09.739583 8955 ts_tablet_manager.cc:1398] T 1fc0c640a6fb4b24b17642ed26ce8f07 P 201d2b6485084e7dad2cbb722af94779: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:09.740303 8954 ts_tablet_manager.cc:1398] T 1fc0c640a6fb4b24b17642ed26ce8f07 P a47e5c56f13843018ac584914a8d4707: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:09.741066 8953 tablet_bootstrap.cc:492] T 1fc0c640a6fb4b24b17642ed26ce8f07 P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:46:09.741230 8953 ts_tablet_manager.cc:1367] T 1fc0c640a6fb4b24b17642ed26ce8f07 P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 20:46:09.741673 8953 raft_consensus.cc:348] T 1fc0c640a6fb4b24b17642ed26ce8f07 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:09.741801 8953 raft_consensus.cc:374] T 1fc0c640a6fb4b24b17642ed26ce8f07 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:09.741833 8953 raft_consensus.cc:729] T 1fc0c640a6fb4b24b17642ed26ce8f07 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:46:09.742102 8953 consensus_queue.cc:260] T 1fc0c640a6fb4b24b17642ed26ce8f07 P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:09.742403 8953 ts_tablet_manager.cc:1398] T 1fc0c640a6fb4b24b17642ed26ce8f07 P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:09.744817 8528 consensus_queue.cc:1035] T ef4002126e8847809add182f262d62d9 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:09.749372 8965 consensus_queue.cc:1035] T ef4002126e8847809add182f262d62d9 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:09.771656 6375 raft_consensus.cc:1050] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc: attempting to promote NON_VOTER 7845c397230d49358ba2fad9db53f97c to VOTER
I20240502 20:46:09.772110 6375 consensus_queue.cc:237] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } }
I20240502 20:46:09.773475 4769 raft_consensus.cc:1261] T d6582234196c4fe4a82b3182787700de P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:09.773869 6702 raft_consensus.cc:1261] T d6582234196c4fe4a82b3182787700de P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:09.774061 8907 consensus_queue.cc:1035] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:09.774451 8907 consensus_queue.cc:1035] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:09.774807 4638 raft_consensus.cc:1261] T d6582234196c4fe4a82b3182787700de P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:09.776078 6375 consensus_queue.cc:1035] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:09.777052 6375 raft_consensus.cc:2932] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:09.778337 4638 raft_consensus.cc:2932] T d6582234196c4fe4a82b3182787700de P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:09.778863 6702 raft_consensus.cc:2932] T d6582234196c4fe4a82b3182787700de P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:09.780421 4769 raft_consensus.cc:2932] T d6582234196c4fe4a82b3182787700de P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:09.780798 4233 catalog_manager.cc:5484] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: config changed from index 4 to 5, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:09.784392 4506 consensus_queue.cc:237] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } }
I20240502 20:46:09.788666 6702 raft_consensus.cc:1261] T d6582234196c4fe4a82b3182787700de P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:09.790145 7967 consensus_queue.cc:1035] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:09.792593 4769 raft_consensus.cc:1261] T d6582234196c4fe4a82b3182787700de P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:09.793540 7967 consensus_queue.cc:1035] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:09.794526 7967 raft_consensus.cc:2932] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:09.794934 6703 raft_consensus.cc:2932] T d6582234196c4fe4a82b3182787700de P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:09.796893 4221 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d6582234196c4fe4a82b3182787700de with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:09.797113 4769 raft_consensus.cc:2932] T d6582234196c4fe4a82b3182787700de P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:09.797473 4232 catalog_manager.cc:5484] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: config changed from index 5 to 6, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:09.800824 4618 tablet_service.cc:1514] Processing DeleteTablet for tablet d6582234196c4fe4a82b3182787700de with delete_type TABLET_DATA_TOMBSTONED (TS 923cfab926e7443f8ffd7b21ad0a9d91 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57068
I20240502 20:46:09.801085 8510 tablet_replica.cc:330] T d6582234196c4fe4a82b3182787700de P 923cfab926e7443f8ffd7b21ad0a9d91: stopping tablet replica
I20240502 20:46:09.801239 8510 raft_consensus.cc:2229] T d6582234196c4fe4a82b3182787700de P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:09.801421 8510 raft_consensus.cc:2258] T d6582234196c4fe4a82b3182787700de P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:09.802258 8510 ts_tablet_manager.cc:1875] T d6582234196c4fe4a82b3182787700de P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:09.804145 8510 ts_tablet_manager.cc:1888] T d6582234196c4fe4a82b3182787700de P 923cfab926e7443f8ffd7b21ad0a9d91: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:46:09.804198 8510 log.cc:1196] T d6582234196c4fe4a82b3182787700de P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal/wals/d6582234196c4fe4a82b3182787700de
I20240502 20:46:09.804656 4219 catalog_manager.cc:4830] TS 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141): tablet d6582234196c4fe4a82b3182787700de (table rebalance_test_table_7 [id=1e32a0fff569475cbd44b12c4fd5e8e6]) successfully deleted
I20240502 20:46:09.833016 4233 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240502 20:46:09.833173 4233 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240502 20:46:09.834849 8857 raft_consensus.cc:1050] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833: attempting to promote NON_VOTER 7845c397230d49358ba2fad9db53f97c to VOTER
I20240502 20:46:09.835318 8857 consensus_queue.cc:237] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } }
I20240502 20:46:09.836017 4233 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: Sending DeleteTablet for 3 replicas of tablet 5a36bf6e8e6e43b2b47b6d6357b0d445
I20240502 20:46:09.836351 4233 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: Sending DeleteTablet for 3 replicas of tablet f09dcfea3eb347e687f3c72f4ccc6929
I20240502 20:46:09.836614 4233 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: Sending DeleteTablet for 3 replicas of tablet 628c976198d64c26ba1d68492416772a
I20240502 20:46:09.837015 4769 raft_consensus.cc:1261] T 180902058815460189f75de4704d1e84 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer a4bf1df7fa6b4948be502c182bfbb833: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:09.837046 4618 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a36bf6e8e6e43b2b47b6d6357b0d445 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:09 UTC) from {username='slave'} at 127.0.0.1:57068
I20240502 20:46:09.837080 7079 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a36bf6e8e6e43b2b47b6d6357b0d445 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:09 UTC) from {username='slave'} at 127.0.0.1:59162
I20240502 20:46:09.837293 8510 tablet_replica.cc:330] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 923cfab926e7443f8ffd7b21ad0a9d91: stopping tablet replica
I20240502 20:46:09.837399 7099 raft_consensus.cc:1261] T 180902058815460189f75de4704d1e84 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Refusing update from remote peer a4bf1df7fa6b4948be502c182bfbb833: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:09.837430 8510 raft_consensus.cc:2229] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:09.837471 4618 tablet_service.cc:1514] Processing DeleteTablet for tablet 628c976198d64c26ba1d68492416772a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:09 UTC) from {username='slave'} at 127.0.0.1:57068
I20240502 20:46:09.837612 8510 raft_consensus.cc:2258] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:09.837810 8364 consensus_queue.cc:1035] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:46:09.837889 8805 tablet_replica.cc:330] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 201d2b6485084e7dad2cbb722af94779: stopping tablet replica
I20240502 20:46:09.837903 4488 tablet_service.cc:1514] Processing DeleteTablet for tablet 628c976198d64c26ba1d68492416772a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:09 UTC) from {username='slave'} at 127.0.0.1:48890
I20240502 20:46:09.838016 8805 raft_consensus.cc:2229] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 201d2b6485084e7dad2cbb722af94779 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:09.838176 8805 raft_consensus.cc:2258] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 201d2b6485084e7dad2cbb722af94779 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:09.838227 8141 tablet_replica.cc:330] T 628c976198d64c26ba1d68492416772a P c3d26eafd51547acb780d4aac5ebe7cc: stopping tablet replica
I20240502 20:46:09.838341 4487 tablet_service.cc:1514] Processing DeleteTablet for tablet f09dcfea3eb347e687f3c72f4ccc6929 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:09 UTC) from {username='slave'} at 127.0.0.1:48890
I20240502 20:46:09.838366 8141 raft_consensus.cc:2229] T 628c976198d64c26ba1d68492416772a P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:09.838985 8189 consensus_queue.cc:1035] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:46:09.839048 8510 ts_tablet_manager.cc:1875] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:09.839048 8805 ts_tablet_manager.cc:1875] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 201d2b6485084e7dad2cbb722af94779: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:09.839147 8141 raft_consensus.cc:2258] T 628c976198d64c26ba1d68492416772a P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:09.839722 8141 ts_tablet_manager.cc:1875] T 628c976198d64c26ba1d68492416772a P c3d26eafd51547acb780d4aac5ebe7cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:09.840256 4354 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a36bf6e8e6e43b2b47b6d6357b0d445 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:09 UTC) from {username='slave'} at 127.0.0.1:38306
I20240502 20:46:09.840801 4750 tablet_service.cc:1514] Processing DeleteTablet for tablet 628c976198d64c26ba1d68492416772a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:09 UTC) from {username='slave'} at 127.0.0.1:38526
I20240502 20:46:09.841290 8510 ts_tablet_manager.cc:1888] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 923cfab926e7443f8ffd7b21ad0a9d91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.7
I20240502 20:46:09.841290 8805 ts_tablet_manager.cc:1888] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 201d2b6485084e7dad2cbb722af94779: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.7
I20240502 20:46:09.841367 8510 log.cc:1196] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal/wals/5a36bf6e8e6e43b2b47b6d6357b0d445
I20240502 20:46:09.841367 8805 log.cc:1196] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 201d2b6485084e7dad2cbb722af94779: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/wal/wals/5a36bf6e8e6e43b2b47b6d6357b0d445
I20240502 20:46:09.841634 8510 ts_tablet_manager.cc:1909] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting consensus metadata
I20240502 20:46:09.841640 8805 ts_tablet_manager.cc:1909] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P 201d2b6485084e7dad2cbb722af94779: Deleting consensus metadata
I20240502 20:46:09.841846 4356 tablet_service.cc:1514] Processing DeleteTablet for tablet f09dcfea3eb347e687f3c72f4ccc6929 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:09 UTC) from {username='slave'} at 127.0.0.1:38306
I20240502 20:46:09.842031 7299 tablet_replica.cc:330] T 628c976198d64c26ba1d68492416772a P ed47c023452a497192f35d2f617bf9bd: stopping tablet replica
I20240502 20:46:09.842048 6947 tablet_service.cc:1514] Processing DeleteTablet for tablet f09dcfea3eb347e687f3c72f4ccc6929 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:09 UTC) from {username='slave'} at 127.0.0.1:45156
I20240502 20:46:09.842162 7299 raft_consensus.cc:2229] T 628c976198d64c26ba1d68492416772a P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:09.842327 7297 tablet_replica.cc:330] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c92e6e46782849479c8a0c6d82276fb2: stopping tablet replica
I20240502 20:46:09.842321 7299 raft_consensus.cc:2258] T 628c976198d64c26ba1d68492416772a P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:09.842376 9035 tablet_replica.cc:330] T f09dcfea3eb347e687f3c72f4ccc6929 P a47e5c56f13843018ac584914a8d4707: stopping tablet replica
I20240502 20:46:09.842460 7297 raft_consensus.cc:2229] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c92e6e46782849479c8a0c6d82276fb2 [term 2 LEADER]: Raft consensus shutting down.
I20240502 20:46:09.842526 9035 raft_consensus.cc:2229] T f09dcfea3eb347e687f3c72f4ccc6929 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:09.842669 7297 raft_consensus.cc:2258] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c92e6e46782849479c8a0c6d82276fb2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:09.842679 9035 raft_consensus.cc:2258] T f09dcfea3eb347e687f3c72f4ccc6929 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:09.842849 7299 ts_tablet_manager.cc:1875] T 628c976198d64c26ba1d68492416772a P ed47c023452a497192f35d2f617bf9bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:09.843168 9035 ts_tablet_manager.cc:1875] T f09dcfea3eb347e687f3c72f4ccc6929 P a47e5c56f13843018ac584914a8d4707: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:09.843705 6703 raft_consensus.cc:1261] T 180902058815460189f75de4704d1e84 P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Refusing update from remote peer a4bf1df7fa6b4948be502c182bfbb833: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:09.844159 9033 consensus_queue.cc:1035] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:46:09.845008 7299 ts_tablet_manager.cc:1888] T 628c976198d64c26ba1d68492416772a P ed47c023452a497192f35d2f617bf9bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 20:46:09.845072 7299 log.cc:1196] T 628c976198d64c26ba1d68492416772a P ed47c023452a497192f35d2f617bf9bd: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal/wals/628c976198d64c26ba1d68492416772a
I20240502 20:46:09.845088 4638 consensus_queue.cc:237] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } }
I20240502 20:46:09.845335 7299 ts_tablet_manager.cc:1909] T 628c976198d64c26ba1d68492416772a P ed47c023452a497192f35d2f617bf9bd: Deleting consensus metadata
I20240502 20:46:09.846141 4222 catalog_manager.cc:4830] TS ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193): tablet 628c976198d64c26ba1d68492416772a (table rebalance_test_table_16 [id=dffd2e92869942b9a6f03505945ad3ba]) successfully deleted
I20240502 20:46:09.846493 9035 ts_tablet_manager.cc:1888] T f09dcfea3eb347e687f3c72f4ccc6929 P a47e5c56f13843018ac584914a8d4707: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240502 20:46:09.846592 9035 log.cc:1196] T f09dcfea3eb347e687f3c72f4ccc6929 P a47e5c56f13843018ac584914a8d4707: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/wal/wals/f09dcfea3eb347e687f3c72f4ccc6929
I20240502 20:46:09.846896 9035 ts_tablet_manager.cc:1909] T f09dcfea3eb347e687f3c72f4ccc6929 P a47e5c56f13843018ac584914a8d4707: Deleting consensus metadata
I20240502 20:46:09.847504 4222 catalog_manager.cc:4830] TS a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): tablet f09dcfea3eb347e687f3c72f4ccc6929 (table rebalance_test_table_16 [id=dffd2e92869942b9a6f03505945ad3ba]) successfully deleted
I20240502 20:46:09.847594 6967 raft_consensus.cc:1261] T 8565ebbd21dd4f708a0254d1ed2d6670 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:09.848258 7297 ts_tablet_manager.cc:1875] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c92e6e46782849479c8a0c6d82276fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:09.849355 4219 catalog_manager.cc:4830] TS 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): tablet 5a36bf6e8e6e43b2b47b6d6357b0d445 (table rebalance_test_table_16 [id=dffd2e92869942b9a6f03505945ad3ba]) successfully deleted
W20240502 20:46:09.849819 4574 consensus_peers.cc:485] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8565ebbd21dd4f708a0254d1ed2d6670. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:09.849637 4507 raft_consensus.cc:1261] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:09.842872 8141 ts_tablet_manager.cc:1888] T 628c976198d64c26ba1d68492416772a P c3d26eafd51547acb780d4aac5ebe7cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 20:46:09.849973 8141 log.cc:1196] T 628c976198d64c26ba1d68492416772a P c3d26eafd51547acb780d4aac5ebe7cc: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal/wals/628c976198d64c26ba1d68492416772a
I20240502 20:46:09.850193 7297 ts_tablet_manager.cc:1888] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c92e6e46782849479c8a0c6d82276fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.7
I20240502 20:46:09.850276 7297 log.cc:1196] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c92e6e46782849479c8a0c6d82276fb2: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal/wals/5a36bf6e8e6e43b2b47b6d6357b0d445
I20240502 20:46:09.850256 8236 consensus_queue.cc:1035] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:09.850512 7297 ts_tablet_manager.cc:1909] T 5a36bf6e8e6e43b2b47b6d6357b0d445 P c92e6e46782849479c8a0c6d82276fb2: Deleting consensus metadata
I20240502 20:46:09.851024 7297 tablet_replica.cc:330] T f09dcfea3eb347e687f3c72f4ccc6929 P c92e6e46782849479c8a0c6d82276fb2: stopping tablet replica
I20240502 20:46:09.851531 8141 ts_tablet_manager.cc:1909] T 628c976198d64c26ba1d68492416772a P c3d26eafd51547acb780d4aac5ebe7cc: Deleting consensus metadata
I20240502 20:46:09.851830 4222 catalog_manager.cc:4830] TS c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827): tablet 5a36bf6e8e6e43b2b47b6d6357b0d445 (table rebalance_test_table_16 [id=dffd2e92869942b9a6f03505945ad3ba]) successfully deleted
I20240502 20:46:09.851853 8236 consensus_queue.cc:1035] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:09.852001 7297 raft_consensus.cc:2229] T f09dcfea3eb347e687f3c72f4ccc6929 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:09.852149 7297 raft_consensus.cc:2258] T f09dcfea3eb347e687f3c72f4ccc6929 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:09.852962 8141 tablet_replica.cc:330] T f09dcfea3eb347e687f3c72f4ccc6929 P c3d26eafd51547acb780d4aac5ebe7cc: stopping tablet replica
I20240502 20:46:09.852931 9033 raft_consensus.cc:2932] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:09.853312 8510 tablet_replica.cc:330] T 628c976198d64c26ba1d68492416772a P 923cfab926e7443f8ffd7b21ad0a9d91: stopping tablet replica
I20240502 20:46:09.853397 7297 ts_tablet_manager.cc:1875] T f09dcfea3eb347e687f3c72f4ccc6929 P c92e6e46782849479c8a0c6d82276fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:09.853498 8510 raft_consensus.cc:2229] T 628c976198d64c26ba1d68492416772a P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:09.853652 8510 raft_consensus.cc:2258] T 628c976198d64c26ba1d68492416772a P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:09.854149 8510 ts_tablet_manager.cc:1875] T 628c976198d64c26ba1d68492416772a P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:09.854931 4221 catalog_manager.cc:4830] TS c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097): tablet 628c976198d64c26ba1d68492416772a (table rebalance_test_table_16 [id=dffd2e92869942b9a6f03505945ad3ba]) successfully deleted
I20240502 20:46:09.855607 7297 ts_tablet_manager.cc:1888] T f09dcfea3eb347e687f3c72f4ccc6929 P c92e6e46782849479c8a0c6d82276fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240502 20:46:09.855682 7297 log.cc:1196] T f09dcfea3eb347e687f3c72f4ccc6929 P c92e6e46782849479c8a0c6d82276fb2: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal/wals/f09dcfea3eb347e687f3c72f4ccc6929
I20240502 20:46:09.855957 7297 ts_tablet_manager.cc:1909] T f09dcfea3eb347e687f3c72f4ccc6929 P c92e6e46782849479c8a0c6d82276fb2: Deleting consensus metadata
I20240502 20:46:09.855958 4769 raft_consensus.cc:2932] T 180902058815460189f75de4704d1e84 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:09.856173 8236 raft_consensus.cc:2932] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } } }
I20240502 20:46:09.857334 6703 raft_consensus.cc:2932] T 180902058815460189f75de4704d1e84 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:09.858037 4219 catalog_manager.cc:4830] TS 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141): tablet 5a36bf6e8e6e43b2b47b6d6357b0d445 (table rebalance_test_table_16 [id=dffd2e92869942b9a6f03505945ad3ba]) successfully deleted
I20240502 20:46:09.859898 4222 catalog_manager.cc:4830] TS c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827): tablet f09dcfea3eb347e687f3c72f4ccc6929 (table rebalance_test_table_16 [id=dffd2e92869942b9a6f03505945ad3ba]) successfully deleted
I20240502 20:46:09.860148 7099 raft_consensus.cc:2932] T 180902058815460189f75de4704d1e84 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:09.860256 4227 catalog_manager.cc:5484] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index 6 to 7, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:09.860224 4232 catalog_manager.cc:5484] T 180902058815460189f75de4704d1e84 P 7845c397230d49358ba2fad9db53f97c reported cstate change: config changed from index 2 to 3, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a4bf1df7fa6b4948be502c182bfbb833" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:09.861851 8510 ts_tablet_manager.cc:1888] T 628c976198d64c26ba1d68492416772a P 923cfab926e7443f8ffd7b21ad0a9d91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 20:46:09.861959 8510 log.cc:1196] T 628c976198d64c26ba1d68492416772a P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal/wals/628c976198d64c26ba1d68492416772a
I20240502 20:46:09.862267 8510 ts_tablet_manager.cc:1909] T 628c976198d64c26ba1d68492416772a P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting consensus metadata
I20240502 20:46:09.862383 6967 raft_consensus.cc:2932] T 8565ebbd21dd4f708a0254d1ed2d6670 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } } }
I20240502 20:46:09.863721 7099 raft_consensus.cc:1203] T eebddad3efe748f78372901ecad5b407 P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:09.865420 4233 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41812:
name: "rebalancer_extra_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:46:09.866624 6835 consensus_queue.cc:237] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } }
I20240502 20:46:09.867852 4219 catalog_manager.cc:4830] TS 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141): tablet 628c976198d64c26ba1d68492416772a (table rebalance_test_table_16 [id=dffd2e92869942b9a6f03505945ad3ba]) successfully deleted
I20240502 20:46:09.869262 8168 raft_consensus.cc:1050] T ed6386b6777c4d50b29ff8cd49894eba P 923cfab926e7443f8ffd7b21ad0a9d91: attempting to promote NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 to VOTER
I20240502 20:46:09.869311 4769 raft_consensus.cc:1261] T 180902058815460189f75de4704d1e84 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer a4bf1df7fa6b4948be502c182bfbb833: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:09.870395 6967 raft_consensus.cc:1203] T c55ba788f592448d8941ff94a8cefc52 P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:09.871156 6703 raft_consensus.cc:1261] T 180902058815460189f75de4704d1e84 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Refusing update from remote peer a4bf1df7fa6b4948be502c182bfbb833: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:09.871505 6831 raft_consensus.cc:1203] T 45ade8c672d8445eaffb179772e3759a P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:09.872025 7937 consensus_queue.cc:1035] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:09.872756 8857 consensus_queue.cc:1035] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:09.869702 8168 consensus_queue.cc:237] T ed6386b6777c4d50b29ff8cd49894eba P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } }
I20240502 20:46:09.875335 4769 raft_consensus.cc:1261] T ed6386b6777c4d50b29ff8cd49894eba P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:09.875540 6835 raft_consensus.cc:1261] T ed6386b6777c4d50b29ff8cd49894eba P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEARNER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:09.875855 9033 consensus_queue.cc:1035] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:09.876058 7017 consensus_queue.cc:1035] T ed6386b6777c4d50b29ff8cd49894eba P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:09.878347 7937 consensus_queue.cc:1035] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:09.878443 9033 raft_consensus.cc:2932] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:09.880041 8168 consensus_queue.cc:1035] T ed6386b6777c4d50b29ff8cd49894eba P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:09.880488 4220 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 180902058815460189f75de4704d1e84 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:09.880945 4769 raft_consensus.cc:2932] T 180902058815460189f75de4704d1e84 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:09.883329 6815 tablet_service.cc:1467] Processing CreateTablet for tablet a9afb75bfc114f59850eaea084a306f0 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=c8bf855c4f414063abc8d3c0e2880292]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:09.883394 7079 tablet_service.cc:1467] Processing CreateTablet for tablet ac57ad308b174217ac74c4cb5e1c7285 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=c8bf855c4f414063abc8d3c0e2880292]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:09.883607 6815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9afb75bfc114f59850eaea084a306f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:09.883639 7079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac57ad308b174217ac74c4cb5e1c7285. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:09.883965 6703 raft_consensus.cc:2932] T 180902058815460189f75de4704d1e84 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:09.886128 6682 tablet_service.cc:1467] Processing CreateTablet for tablet a9afb75bfc114f59850eaea084a306f0 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=c8bf855c4f414063abc8d3c0e2880292]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:09.886150 4228 catalog_manager.cc:5484] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833 reported cstate change: config changed from index 3 to 4, VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) evicted. New cstate: current_term: 1 leader_uuid: "a4bf1df7fa6b4948be502c182bfbb833" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:09.886369 6682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9afb75bfc114f59850eaea084a306f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:09.886390 6947 tablet_service.cc:1467] Processing CreateTablet for tablet a9afb75bfc114f59850eaea084a306f0 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=c8bf855c4f414063abc8d3c0e2880292]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:09.886631 6947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9afb75bfc114f59850eaea084a306f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:09.867661 4507 raft_consensus.cc:2932] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } } }
I20240502 20:46:09.877246 4506 raft_consensus.cc:1261] T ed6386b6777c4d50b29ff8cd49894eba P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:09.853159 8141 raft_consensus.cc:2229] T f09dcfea3eb347e687f3c72f4ccc6929 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:09.888650 8954 tablet_bootstrap.cc:492] T a9afb75bfc114f59850eaea084a306f0 P a47e5c56f13843018ac584914a8d4707: Bootstrap starting.
I20240502 20:46:09.888731 6683 tablet_service.cc:1467] Processing CreateTablet for tablet ac57ad308b174217ac74c4cb5e1c7285 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=c8bf855c4f414063abc8d3c0e2880292]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:09.888967 6683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac57ad308b174217ac74c4cb5e1c7285. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:09.889819 8954 tablet_bootstrap.cc:654] T a9afb75bfc114f59850eaea084a306f0 P a47e5c56f13843018ac584914a8d4707: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:09.891036 8954 tablet_bootstrap.cc:492] T a9afb75bfc114f59850eaea084a306f0 P a47e5c56f13843018ac584914a8d4707: No bootstrap required, opened a new log
I20240502 20:46:09.891113 8954 ts_tablet_manager.cc:1367] T a9afb75bfc114f59850eaea084a306f0 P a47e5c56f13843018ac584914a8d4707: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:46:09.891074 6681 tablet_service.cc:1467] Processing CreateTablet for tablet 77dc51e6a85a4ddcb64db12b40c5810f (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=c8bf855c4f414063abc8d3c0e2880292]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:09.891336 6681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77dc51e6a85a4ddcb64db12b40c5810f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:09.891577 8954 raft_consensus.cc:348] T a9afb75bfc114f59850eaea084a306f0 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:09.891676 8954 raft_consensus.cc:374] T a9afb75bfc114f59850eaea084a306f0 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:09.891708 8954 raft_consensus.cc:729] T a9afb75bfc114f59850eaea084a306f0 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Initialized, Role: FOLLOWER
I20240502 20:46:09.891825 8954 consensus_queue.cc:260] T a9afb75bfc114f59850eaea084a306f0 P a47e5c56f13843018ac584914a8d4707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:09.892102 8954 ts_tablet_manager.cc:1398] T a9afb75bfc114f59850eaea084a306f0 P a47e5c56f13843018ac584914a8d4707: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:09.888875 8141 raft_consensus.cc:2258] T f09dcfea3eb347e687f3c72f4ccc6929 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:09.892810 8236 raft_consensus.cc:2932] T ed6386b6777c4d50b29ff8cd49894eba P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:09.893273 8141 ts_tablet_manager.cc:1875] T f09dcfea3eb347e687f3c72f4ccc6929 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:09.894603 7078 tablet_service.cc:1514] Processing DeleteTablet for tablet 180902058815460189f75de4704d1e84 with delete_type TABLET_DATA_TOMBSTONED (TS 201d2b6485084e7dad2cbb722af94779 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59162
I20240502 20:46:09.894862 8805 tablet_replica.cc:330] T 180902058815460189f75de4704d1e84 P 201d2b6485084e7dad2cbb722af94779: stopping tablet replica
I20240502 20:46:09.894984 8805 raft_consensus.cc:2229] T 180902058815460189f75de4704d1e84 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:09.895102 8805 raft_consensus.cc:2258] T 180902058815460189f75de4704d1e84 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:09.896253 4506 raft_consensus.cc:1203] T 747e7331d7ad4dd4b132672254037e1b P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 20:46:09.885535 4487 tablet_service.cc:1467] Processing CreateTablet for tablet ac57ad308b174217ac74c4cb5e1c7285 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=c8bf855c4f414063abc8d3c0e2880292]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:09.896567 4487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac57ad308b174217ac74c4cb5e1c7285. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:09.898360 4768 raft_consensus.cc:2932] T ed6386b6777c4d50b29ff8cd49894eba P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:09.886106 4488 tablet_service.cc:1467] Processing CreateTablet for tablet 77dc51e6a85a4ddcb64db12b40c5810f (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=c8bf855c4f414063abc8d3c0e2880292]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:09.898768 6835 raft_consensus.cc:2932] T ed6386b6777c4d50b29ff8cd49894eba P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:09.898953 4233 catalog_manager.cc:5484] T ed6386b6777c4d50b29ff8cd49894eba P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index 4 to 5, a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:09.899091 4488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77dc51e6a85a4ddcb64db12b40c5810f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:09.902613 8805 ts_tablet_manager.cc:1875] T 180902058815460189f75de4704d1e84 P 201d2b6485084e7dad2cbb722af94779: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:09.905193 8965 raft_consensus.cc:482] T 1fc0c640a6fb4b24b17642ed26ce8f07 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:09.905294 8965 raft_consensus.cc:504] T 1fc0c640a6fb4b24b17642ed26ce8f07 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:09.905731 8965 leader_election.cc:288] T 1fc0c640a6fb4b24b17642ed26ce8f07 P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
I20240502 20:46:09.906682 7099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc0c640a6fb4b24b17642ed26ce8f07" candidate_uuid: "a47e5c56f13843018ac584914a8d4707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201d2b6485084e7dad2cbb722af94779" is_pre_election: true
I20240502 20:46:09.906716 4618 tablet_service.cc:1467] Processing CreateTablet for tablet 77dc51e6a85a4ddcb64db12b40c5810f (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=c8bf855c4f414063abc8d3c0e2880292]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:09.906813 7099 raft_consensus.cc:2454] T 1fc0c640a6fb4b24b17642ed26ce8f07 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a47e5c56f13843018ac584914a8d4707 in term 0.
I20240502 20:46:09.907012 4618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77dc51e6a85a4ddcb64db12b40c5810f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:09.907351 4638 consensus_queue.cc:237] T ed6386b6777c4d50b29ff8cd49894eba P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } }
I20240502 20:46:09.907673 6900 leader_election.cc:302] T 1fc0c640a6fb4b24b17642ed26ce8f07 P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201d2b6485084e7dad2cbb722af94779, a47e5c56f13843018ac584914a8d4707; no voters:
I20240502 20:46:09.908015 8955 tablet_bootstrap.cc:492] T ac57ad308b174217ac74c4cb5e1c7285 P 201d2b6485084e7dad2cbb722af94779: Bootstrap starting.
I20240502 20:46:09.908085 8965 raft_consensus.cc:2781] T 1fc0c640a6fb4b24b17642ed26ce8f07 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:09.908154 8965 raft_consensus.cc:482] T 1fc0c640a6fb4b24b17642ed26ce8f07 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:09.908192 8965 raft_consensus.cc:3037] T 1fc0c640a6fb4b24b17642ed26ce8f07 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:09.908638 6835 raft_consensus.cc:1261] T ed6386b6777c4d50b29ff8cd49894eba P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:09.909021 8955 tablet_bootstrap.cc:654] T ac57ad308b174217ac74c4cb5e1c7285 P 201d2b6485084e7dad2cbb722af94779: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:09.909058 7017 consensus_queue.cc:1035] T ed6386b6777c4d50b29ff8cd49894eba P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:09.909229 8965 raft_consensus.cc:504] T 1fc0c640a6fb4b24b17642ed26ce8f07 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:09.909608 8965 leader_election.cc:288] T 1fc0c640a6fb4b24b17642ed26ce8f07 P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 1 election: Requested vote from peers ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
I20240502 20:46:09.910061 7099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc0c640a6fb4b24b17642ed26ce8f07" candidate_uuid: "a47e5c56f13843018ac584914a8d4707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201d2b6485084e7dad2cbb722af94779"
I20240502 20:46:09.910166 7099 raft_consensus.cc:3037] T 1fc0c640a6fb4b24b17642ed26ce8f07 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:09.911032 7099 raft_consensus.cc:2454] T 1fc0c640a6fb4b24b17642ed26ce8f07 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a47e5c56f13843018ac584914a8d4707 in term 1.
I20240502 20:46:09.911259 4768 raft_consensus.cc:1261] T ed6386b6777c4d50b29ff8cd49894eba P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:09.911345 6900 leader_election.cc:302] T 1fc0c640a6fb4b24b17642ed26ce8f07 P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201d2b6485084e7dad2cbb722af94779, a47e5c56f13843018ac584914a8d4707; no voters:
I20240502 20:46:09.911510 8965 raft_consensus.cc:2781] T 1fc0c640a6fb4b24b17642ed26ce8f07 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:09.911568 8965 raft_consensus.cc:686] T 1fc0c640a6fb4b24b17642ed26ce8f07 P a47e5c56f13843018ac584914a8d4707 [term 1 LEADER]: Becoming Leader. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Running, Role: LEADER
I20240502 20:46:09.911679 8965 consensus_queue.cc:237] T 1fc0c640a6fb4b24b17642ed26ce8f07 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:09.911722 8168 consensus_queue.cc:1035] T ed6386b6777c4d50b29ff8cd49894eba P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:09.904942 8805 ts_tablet_manager.cc:1888] T 180902058815460189f75de4704d1e84 P 201d2b6485084e7dad2cbb722af94779: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 20:46:09.912034 8805 log.cc:1196] T 180902058815460189f75de4704d1e84 P 201d2b6485084e7dad2cbb722af94779: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/wal/wals/180902058815460189f75de4704d1e84
I20240502 20:46:09.912741 4219 catalog_manager.cc:4830] TS 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): tablet 180902058815460189f75de4704d1e84 (table rebalancer_extra_table_2 [id=44d455c601714afb8333ce2027e68a4a]) successfully deleted
I20240502 20:46:09.913949 8141 ts_tablet_manager.cc:1888] T f09dcfea3eb347e687f3c72f4ccc6929 P c3d26eafd51547acb780d4aac5ebe7cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240502 20:46:09.914026 8141 log.cc:1196] T f09dcfea3eb347e687f3c72f4ccc6929 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal/wals/f09dcfea3eb347e687f3c72f4ccc6929
I20240502 20:46:09.914033 9056 tablet_bootstrap.cc:492] T a9afb75bfc114f59850eaea084a306f0 P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap starting.
I20240502 20:46:09.914294 8141 ts_tablet_manager.cc:1909] T f09dcfea3eb347e687f3c72f4ccc6929 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting consensus metadata
I20240502 20:46:09.915197 9056 tablet_bootstrap.cc:654] T a9afb75bfc114f59850eaea084a306f0 P a4bf1df7fa6b4948be502c182bfbb833: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:09.915180 4227 catalog_manager.cc:5484] T 1fc0c640a6fb4b24b17642ed26ce8f07 P a47e5c56f13843018ac584914a8d4707 reported cstate change: term changed from 0 to 1, leader changed from <none> to a47e5c56f13843018ac584914a8d4707 (127.30.32.71). New cstate: current_term: 1 leader_uuid: "a47e5c56f13843018ac584914a8d4707" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:09.915448 9057 tablet_bootstrap.cc:492] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c: Bootstrap starting.
I20240502 20:46:09.915375 8236 raft_consensus.cc:2932] T ed6386b6777c4d50b29ff8cd49894eba P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:09.915825 4221 catalog_manager.cc:4830] TS c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097): tablet f09dcfea3eb347e687f3c72f4ccc6929 (table rebalance_test_table_16 [id=dffd2e92869942b9a6f03505945ad3ba]) successfully deleted
I20240502 20:46:09.917464 8955 tablet_bootstrap.cc:492] T ac57ad308b174217ac74c4cb5e1c7285 P 201d2b6485084e7dad2cbb722af94779: No bootstrap required, opened a new log
I20240502 20:46:09.917567 8955 ts_tablet_manager.cc:1367] T ac57ad308b174217ac74c4cb5e1c7285 P 201d2b6485084e7dad2cbb722af94779: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20240502 20:46:09.918083 8955 raft_consensus.cc:348] T ac57ad308b174217ac74c4cb5e1c7285 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:09.918172 8955 raft_consensus.cc:374] T ac57ad308b174217ac74c4cb5e1c7285 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:09.918210 8955 raft_consensus.cc:729] T ac57ad308b174217ac74c4cb5e1c7285 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Initialized, Role: FOLLOWER
I20240502 20:46:09.918352 4219 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ed6386b6777c4d50b29ff8cd49894eba with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:09.918345 8955 consensus_queue.cc:260] T ac57ad308b174217ac74c4cb5e1c7285 P 201d2b6485084e7dad2cbb722af94779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:09.918653 8955 ts_tablet_manager.cc:1398] T ac57ad308b174217ac74c4cb5e1c7285 P 201d2b6485084e7dad2cbb722af94779: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:09.918843 4233 catalog_manager.cc:5484] T ed6386b6777c4d50b29ff8cd49894eba P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index 5 to 6, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:09.919680 4768 raft_consensus.cc:2932] T ed6386b6777c4d50b29ff8cd49894eba P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:09.920917 9060 tablet_bootstrap.cc:492] T ac57ad308b174217ac74c4cb5e1c7285 P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:46:09.921933 9057 tablet_bootstrap.cc:654] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:09.922288 4488 tablet_service.cc:1514] Processing DeleteTablet for tablet ed6386b6777c4d50b29ff8cd49894eba with delete_type TABLET_DATA_TOMBSTONED (TS c3d26eafd51547acb780d4aac5ebe7cc not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48890
I20240502 20:46:09.922554 8141 tablet_replica.cc:330] T ed6386b6777c4d50b29ff8cd49894eba P c3d26eafd51547acb780d4aac5ebe7cc: stopping tablet replica
I20240502 20:46:09.922736 8141 raft_consensus.cc:2229] T ed6386b6777c4d50b29ff8cd49894eba P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:09.922945 8141 raft_consensus.cc:2258] T ed6386b6777c4d50b29ff8cd49894eba P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:09.923380 9062 tablet_bootstrap.cc:492] T 77dc51e6a85a4ddcb64db12b40c5810f P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:46:09.923507 8141 ts_tablet_manager.cc:1875] T ed6386b6777c4d50b29ff8cd49894eba P c3d26eafd51547acb780d4aac5ebe7cc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:09.924489 9062 tablet_bootstrap.cc:654] T 77dc51e6a85a4ddcb64db12b40c5810f P 923cfab926e7443f8ffd7b21ad0a9d91: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:09.917959 6835 raft_consensus.cc:2932] T ed6386b6777c4d50b29ff8cd49894eba P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:09.926957 9062 tablet_bootstrap.cc:492] T 77dc51e6a85a4ddcb64db12b40c5810f P 923cfab926e7443f8ffd7b21ad0a9d91: No bootstrap required, opened a new log
I20240502 20:46:09.927050 9062 ts_tablet_manager.cc:1367] T 77dc51e6a85a4ddcb64db12b40c5810f P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 20:46:09.927578 9062 raft_consensus.cc:348] T 77dc51e6a85a4ddcb64db12b40c5810f P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:09.927677 9062 raft_consensus.cc:374] T 77dc51e6a85a4ddcb64db12b40c5810f P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:09.927716 9062 raft_consensus.cc:729] T 77dc51e6a85a4ddcb64db12b40c5810f P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: FOLLOWER
I20240502 20:46:09.927855 9062 consensus_queue.cc:260] T 77dc51e6a85a4ddcb64db12b40c5810f P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:09.928182 9062 ts_tablet_manager.cc:1398] T 77dc51e6a85a4ddcb64db12b40c5810f P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:09.930922 9060 tablet_bootstrap.cc:654] T ac57ad308b174217ac74c4cb5e1c7285 P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:09.931342 6241 raft_consensus.cc:1050] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd: attempting to promote NON_VOTER 201d2b6485084e7dad2cbb722af94779 to VOTER
I20240502 20:46:09.931545 9056 tablet_bootstrap.cc:492] T a9afb75bfc114f59850eaea084a306f0 P a4bf1df7fa6b4948be502c182bfbb833: No bootstrap required, opened a new log
I20240502 20:46:09.931704 9056 ts_tablet_manager.cc:1367] T a9afb75bfc114f59850eaea084a306f0 P a4bf1df7fa6b4948be502c182bfbb833: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20240502 20:46:09.931762 6241 consensus_queue.cc:237] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } }
I20240502 20:46:09.932372 9056 raft_consensus.cc:348] T a9afb75bfc114f59850eaea084a306f0 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:09.932478 9056 raft_consensus.cc:374] T a9afb75bfc114f59850eaea084a306f0 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:09.932508 9056 raft_consensus.cc:729] T a9afb75bfc114f59850eaea084a306f0 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Initialized, Role: FOLLOWER
I20240502 20:46:09.932684 9056 consensus_queue.cc:260] T a9afb75bfc114f59850eaea084a306f0 P a4bf1df7fa6b4948be502c182bfbb833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:09.933188 8141 ts_tablet_manager.cc:1888] T ed6386b6777c4d50b29ff8cd49894eba P c3d26eafd51547acb780d4aac5ebe7cc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 20:46:09.933262 8141 log.cc:1196] T ed6386b6777c4d50b29ff8cd49894eba P c3d26eafd51547acb780d4aac5ebe7cc: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal/wals/ed6386b6777c4d50b29ff8cd49894eba
I20240502 20:46:09.933619 4374 raft_consensus.cc:1261] T cdf218d5e45b4db19a8e2c22dbc5cc5d P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:09.933861 4221 catalog_manager.cc:4830] TS c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097): tablet ed6386b6777c4d50b29ff8cd49894eba (table rebalance_test_table_9 [id=7c90eeb05db84afb8f78ab7f14e0ced4]) successfully deleted
I20240502 20:46:09.934298 4638 raft_consensus.cc:1261] T cdf218d5e45b4db19a8e2c22dbc5cc5d P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:09.934932 8111 consensus_queue.cc:1035] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:09.935590 9056 ts_tablet_manager.cc:1398] T a9afb75bfc114f59850eaea084a306f0 P a4bf1df7fa6b4948be502c182bfbb833: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 20:46:09.935806 7099 raft_consensus.cc:1261] T cdf218d5e45b4db19a8e2c22dbc5cc5d P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:09.935885 9069 consensus_queue.cc:1035] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:09.937153 9057 tablet_bootstrap.cc:492] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c: No bootstrap required, opened a new log
I20240502 20:46:09.937263 9057 ts_tablet_manager.cc:1367] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240502 20:46:09.937736 9057 raft_consensus.cc:348] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:09.937810 8111 consensus_queue.cc:1035] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:09.937865 9057 raft_consensus.cc:374] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:09.937897 9057 raft_consensus.cc:729] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Initialized, Role: FOLLOWER
I20240502 20:46:09.938047 9057 consensus_queue.cc:260] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:09.938347 9057 ts_tablet_manager.cc:1398] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:09.938537 9057 tablet_bootstrap.cc:492] T ac57ad308b174217ac74c4cb5e1c7285 P 7845c397230d49358ba2fad9db53f97c: Bootstrap starting.
I20240502 20:46:09.939213 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc0c640a6fb4b24b17642ed26ce8f07" candidate_uuid: "a47e5c56f13843018ac584914a8d4707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd" is_pre_election: true
I20240502 20:46:09.939265 4769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc0c640a6fb4b24b17642ed26ce8f07" candidate_uuid: "a47e5c56f13843018ac584914a8d4707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd"
I20240502 20:46:09.939327 4768 raft_consensus.cc:2454] T 1fc0c640a6fb4b24b17642ed26ce8f07 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a47e5c56f13843018ac584914a8d4707 in term 0.
I20240502 20:46:09.939363 9057 tablet_bootstrap.cc:654] T ac57ad308b174217ac74c4cb5e1c7285 P 7845c397230d49358ba2fad9db53f97c: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:09.941046 9060 tablet_bootstrap.cc:492] T ac57ad308b174217ac74c4cb5e1c7285 P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:46:09.941196 9060 ts_tablet_manager.cc:1367] T ac57ad308b174217ac74c4cb5e1c7285 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.001s
I20240502 20:46:09.941757 9060 raft_consensus.cc:348] T ac57ad308b174217ac74c4cb5e1c7285 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:09.941898 9060 raft_consensus.cc:374] T ac57ad308b174217ac74c4cb5e1c7285 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:09.941955 9060 raft_consensus.cc:729] T ac57ad308b174217ac74c4cb5e1c7285 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:46:09.942111 9060 consensus_queue.cc:260] T ac57ad308b174217ac74c4cb5e1c7285 P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:09.942649 9060 ts_tablet_manager.cc:1398] T ac57ad308b174217ac74c4cb5e1c7285 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:09.942967 9060 tablet_bootstrap.cc:492] T 77dc51e6a85a4ddcb64db12b40c5810f P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:46:09.943953 9060 tablet_bootstrap.cc:654] T 77dc51e6a85a4ddcb64db12b40c5810f P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:09.947165 8111 raft_consensus.cc:2932] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:09.947559 4374 raft_consensus.cc:2932] T cdf218d5e45b4db19a8e2c22dbc5cc5d P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:09.950029 4638 raft_consensus.cc:2932] T cdf218d5e45b4db19a8e2c22dbc5cc5d P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:09.950492 4233 catalog_manager.cc:5484] T cdf218d5e45b4db19a8e2c22dbc5cc5d P c92e6e46782849479c8a0c6d82276fb2 reported cstate change: config changed from index 4 to 5, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:09.958127 9057 tablet_bootstrap.cc:492] T ac57ad308b174217ac74c4cb5e1c7285 P 7845c397230d49358ba2fad9db53f97c: No bootstrap required, opened a new log
I20240502 20:46:09.958253 9057 ts_tablet_manager.cc:1367] T ac57ad308b174217ac74c4cb5e1c7285 P 7845c397230d49358ba2fad9db53f97c: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.000s
I20240502 20:46:09.958717 9057 raft_consensus.cc:348] T ac57ad308b174217ac74c4cb5e1c7285 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:09.958842 9057 raft_consensus.cc:374] T ac57ad308b174217ac74c4cb5e1c7285 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:09.958882 9057 raft_consensus.cc:729] T ac57ad308b174217ac74c4cb5e1c7285 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Initialized, Role: FOLLOWER
I20240502 20:46:09.958998 9057 consensus_queue.cc:260] T ac57ad308b174217ac74c4cb5e1c7285 P 7845c397230d49358ba2fad9db53f97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:09.959278 9057 ts_tablet_manager.cc:1398] T ac57ad308b174217ac74c4cb5e1c7285 P 7845c397230d49358ba2fad9db53f97c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:09.959441 9057 tablet_bootstrap.cc:492] T 77dc51e6a85a4ddcb64db12b40c5810f P 7845c397230d49358ba2fad9db53f97c: Bootstrap starting.
I20240502 20:46:09.960391 9057 tablet_bootstrap.cc:654] T 77dc51e6a85a4ddcb64db12b40c5810f P 7845c397230d49358ba2fad9db53f97c: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:09.961974 9060 tablet_bootstrap.cc:492] T 77dc51e6a85a4ddcb64db12b40c5810f P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:46:09.961997 9057 tablet_bootstrap.cc:492] T 77dc51e6a85a4ddcb64db12b40c5810f P 7845c397230d49358ba2fad9db53f97c: No bootstrap required, opened a new log
I20240502 20:46:09.962086 9057 ts_tablet_manager.cc:1367] T 77dc51e6a85a4ddcb64db12b40c5810f P 7845c397230d49358ba2fad9db53f97c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:46:09.962291 9060 ts_tablet_manager.cc:1367] T 77dc51e6a85a4ddcb64db12b40c5810f P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.001s
I20240502 20:46:09.962641 9057 raft_consensus.cc:348] T 77dc51e6a85a4ddcb64db12b40c5810f P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:09.962775 9057 raft_consensus.cc:374] T 77dc51e6a85a4ddcb64db12b40c5810f P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:09.962823 9057 raft_consensus.cc:729] T 77dc51e6a85a4ddcb64db12b40c5810f P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Initialized, Role: FOLLOWER
I20240502 20:46:09.962960 9057 consensus_queue.cc:260] T 77dc51e6a85a4ddcb64db12b40c5810f P 7845c397230d49358ba2fad9db53f97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:09.962985 9060 raft_consensus.cc:348] T 77dc51e6a85a4ddcb64db12b40c5810f P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:09.963132 9060 raft_consensus.cc:374] T 77dc51e6a85a4ddcb64db12b40c5810f P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:09.963187 9060 raft_consensus.cc:729] T 77dc51e6a85a4ddcb64db12b40c5810f P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:46:09.963297 9057 ts_tablet_manager.cc:1398] T 77dc51e6a85a4ddcb64db12b40c5810f P 7845c397230d49358ba2fad9db53f97c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:09.963275 7099 raft_consensus.cc:2932] T cdf218d5e45b4db19a8e2c22dbc5cc5d P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:09.963439 9060 consensus_queue.cc:260] T 77dc51e6a85a4ddcb64db12b40c5810f P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:09.963757 9060 ts_tablet_manager.cc:1398] T 77dc51e6a85a4ddcb64db12b40c5810f P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:09.964866 4769 consensus_queue.cc:237] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } }
I20240502 20:46:09.966295 7098 raft_consensus.cc:1261] T cdf218d5e45b4db19a8e2c22dbc5cc5d P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:09.966629 4374 raft_consensus.cc:1261] T cdf218d5e45b4db19a8e2c22dbc5cc5d P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:09.967082 9069 consensus_queue.cc:1035] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:09.968154 9069 consensus_queue.cc:1035] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:09.970181 7996 raft_consensus.cc:2932] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:09.970614 4374 raft_consensus.cc:2932] T cdf218d5e45b4db19a8e2c22dbc5cc5d P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:09.972268 4222 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cdf218d5e45b4db19a8e2c22dbc5cc5d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:09.973039 4233 catalog_manager.cc:5484] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd reported cstate change: config changed from index 5 to 6, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:09.975037 7098 raft_consensus.cc:2932] T cdf218d5e45b4db19a8e2c22dbc5cc5d P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:09.976652 4618 tablet_service.cc:1514] Processing DeleteTablet for tablet cdf218d5e45b4db19a8e2c22dbc5cc5d with delete_type TABLET_DATA_TOMBSTONED (TS 923cfab926e7443f8ffd7b21ad0a9d91 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57068
I20240502 20:46:09.976924 8510 tablet_replica.cc:330] T cdf218d5e45b4db19a8e2c22dbc5cc5d P 923cfab926e7443f8ffd7b21ad0a9d91: stopping tablet replica
I20240502 20:46:09.977126 8510 raft_consensus.cc:2229] T cdf218d5e45b4db19a8e2c22dbc5cc5d P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:09.977306 8510 raft_consensus.cc:2258] T cdf218d5e45b4db19a8e2c22dbc5cc5d P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:09.979300 8510 ts_tablet_manager.cc:1875] T cdf218d5e45b4db19a8e2c22dbc5cc5d P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:09.981457 8510 ts_tablet_manager.cc:1888] T cdf218d5e45b4db19a8e2c22dbc5cc5d P 923cfab926e7443f8ffd7b21ad0a9d91: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:46:09.981595 8510 log.cc:1196] T cdf218d5e45b4db19a8e2c22dbc5cc5d P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal/wals/cdf218d5e45b4db19a8e2c22dbc5cc5d
I20240502 20:46:09.982128 4219 catalog_manager.cc:4830] TS 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141): tablet cdf218d5e45b4db19a8e2c22dbc5cc5d (table rebalance_test_table_19 [id=46a52eb51c614964a91ee65db96cf854]) successfully deleted
I20240502 20:46:10.012936 8506 raft_consensus.cc:482] T ac57ad308b174217ac74c4cb5e1c7285 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:10.013121 8506 raft_consensus.cc:504] T ac57ad308b174217ac74c4cb5e1c7285 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:10.013629 8506 leader_election.cc:288] T ac57ad308b174217ac74c4cb5e1c7285 P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
I20240502 20:46:10.014261 4507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac57ad308b174217ac74c4cb5e1c7285" candidate_uuid: "7845c397230d49358ba2fad9db53f97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" is_pre_election: true
I20240502 20:46:10.014415 4507 raft_consensus.cc:2454] T ac57ad308b174217ac74c4cb5e1c7285 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7845c397230d49358ba2fad9db53f97c in term 0.
I20240502 20:46:10.014787 6638 leader_election.cc:302] T ac57ad308b174217ac74c4cb5e1c7285 P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7845c397230d49358ba2fad9db53f97c, c3d26eafd51547acb780d4aac5ebe7cc; no voters:
I20240502 20:46:10.014876 7098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac57ad308b174217ac74c4cb5e1c7285" candidate_uuid: "7845c397230d49358ba2fad9db53f97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201d2b6485084e7dad2cbb722af94779" is_pre_election: true
I20240502 20:46:10.015023 8506 raft_consensus.cc:2781] T ac57ad308b174217ac74c4cb5e1c7285 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:10.015020 7098 raft_consensus.cc:2454] T ac57ad308b174217ac74c4cb5e1c7285 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7845c397230d49358ba2fad9db53f97c in term 0.
I20240502 20:46:10.015079 8506 raft_consensus.cc:482] T ac57ad308b174217ac74c4cb5e1c7285 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:10.015117 8506 raft_consensus.cc:3037] T ac57ad308b174217ac74c4cb5e1c7285 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:10.016222 8506 raft_consensus.cc:504] T ac57ad308b174217ac74c4cb5e1c7285 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:10.016598 8506 leader_election.cc:288] T ac57ad308b174217ac74c4cb5e1c7285 P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 election: Requested vote from peers c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
I20240502 20:46:10.018362 7098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac57ad308b174217ac74c4cb5e1c7285" candidate_uuid: "7845c397230d49358ba2fad9db53f97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201d2b6485084e7dad2cbb722af94779"
I20240502 20:46:10.018479 7098 raft_consensus.cc:3037] T ac57ad308b174217ac74c4cb5e1c7285 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:10.018469 4507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac57ad308b174217ac74c4cb5e1c7285" candidate_uuid: "7845c397230d49358ba2fad9db53f97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc"
I20240502 20:46:10.018589 4507 raft_consensus.cc:3037] T ac57ad308b174217ac74c4cb5e1c7285 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:10.019522 7098 raft_consensus.cc:2454] T ac57ad308b174217ac74c4cb5e1c7285 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7845c397230d49358ba2fad9db53f97c in term 1.
I20240502 20:46:10.019558 4507 raft_consensus.cc:2454] T ac57ad308b174217ac74c4cb5e1c7285 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7845c397230d49358ba2fad9db53f97c in term 1.
I20240502 20:46:10.019933 6636 leader_election.cc:302] T ac57ad308b174217ac74c4cb5e1c7285 P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201d2b6485084e7dad2cbb722af94779, 7845c397230d49358ba2fad9db53f97c; no voters:
I20240502 20:46:10.020291 8506 raft_consensus.cc:2781] T ac57ad308b174217ac74c4cb5e1c7285 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:10.020445 8506 raft_consensus.cc:686] T ac57ad308b174217ac74c4cb5e1c7285 P 7845c397230d49358ba2fad9db53f97c [term 1 LEADER]: Becoming Leader. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Running, Role: LEADER
I20240502 20:46:10.020592 8506 consensus_queue.cc:237] T ac57ad308b174217ac74c4cb5e1c7285 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:10.024691 9041 raft_consensus.cc:1050] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91: attempting to promote NON_VOTER a47e5c56f13843018ac584914a8d4707 to VOTER
I20240502 20:46:10.025120 9041 consensus_queue.cc:237] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } }
I20240502 20:46:10.025115 4230 catalog_manager.cc:5484] T ac57ad308b174217ac74c4cb5e1c7285 P 7845c397230d49358ba2fad9db53f97c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7845c397230d49358ba2fad9db53f97c (127.30.32.69). New cstate: current_term: 1 leader_uuid: "7845c397230d49358ba2fad9db53f97c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:10.027550 4769 raft_consensus.cc:1261] T 0cf7e95cb91b4e74b14365fcc6b7f968 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:10.027930 4374 raft_consensus.cc:1261] T 0cf7e95cb91b4e74b14365fcc6b7f968 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:10.028453 6967 raft_consensus.cc:1261] T 0cf7e95cb91b4e74b14365fcc6b7f968 P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:10.029287 9043 consensus_queue.cc:1035] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:10.029466 8236 consensus_queue.cc:1035] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:10.029721 9042 consensus_queue.cc:1035] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:10.037190 9042 raft_consensus.cc:2932] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:10.037658 4374 raft_consensus.cc:2932] T 0cf7e95cb91b4e74b14365fcc6b7f968 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:10.040422 4228 catalog_manager.cc:5484] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index 4 to 5, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:10.040627 4769 raft_consensus.cc:2932] T 0cf7e95cb91b4e74b14365fcc6b7f968 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:10.041141 6967 raft_consensus.cc:2932] T 0cf7e95cb91b4e74b14365fcc6b7f968 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:10.046101 4638 consensus_queue.cc:237] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } }
I20240502 20:46:10.047601 4230 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 20:46:10.048935 6967 raft_consensus.cc:1261] T 0cf7e95cb91b4e74b14365fcc6b7f968 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:10.050318 4769 raft_consensus.cc:1261] T 0cf7e95cb91b4e74b14365fcc6b7f968 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:10.051183 9043 consensus_queue.cc:1035] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:10.051565 9043 consensus_queue.cc:1035] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:10.056051 7098 raft_consensus.cc:1261] T 1fc0c640a6fb4b24b17642ed26ce8f07 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Refusing update from remote peer a47e5c56f13843018ac584914a8d4707: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:10.056547 8965 consensus_queue.cc:1035] T 1fc0c640a6fb4b24b17642ed26ce8f07 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:10.057389 4769 raft_consensus.cc:3037] T 1fc0c640a6fb4b24b17642ed26ce8f07 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:10.058507 4769 raft_consensus.cc:1261] T 1fc0c640a6fb4b24b17642ed26ce8f07 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer a47e5c56f13843018ac584914a8d4707: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:10.058954 4230 catalog_manager.cc:5576] TS 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643) does not have the latest schema for tablet 1fc0c640a6fb4b24b17642ed26ce8f07 (table rebalancer_dynamic_table [id=44b9d8eb907c483bb3ba9399264a26fb]). Expected version 1 got 0
I20240502 20:46:10.059530 8528 consensus_queue.cc:1035] T 1fc0c640a6fb4b24b17642ed26ce8f07 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:10.062817 4230 catalog_manager.cc:5576] TS 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643) does not have the latest schema for tablet 1fc0c640a6fb4b24b17642ed26ce8f07 (table rebalancer_dynamic_table [id=44b9d8eb907c483bb3ba9399264a26fb]). Expected version 1 got 0
I20240502 20:46:10.063290 8236 raft_consensus.cc:2932] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:10.064618 4230 catalog_manager.cc:5576] TS ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193) does not have the latest schema for tablet 1fc0c640a6fb4b24b17642ed26ce8f07 (table rebalancer_dynamic_table [id=44b9d8eb907c483bb3ba9399264a26fb]). Expected version 1 got 0
I20240502 20:46:10.065179 4219 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0cf7e95cb91b4e74b14365fcc6b7f968 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:10.066568 4231 catalog_manager.cc:5576] TS a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841) does not have the latest schema for tablet 1fc0c640a6fb4b24b17642ed26ce8f07 (table rebalancer_dynamic_table [id=44b9d8eb907c483bb3ba9399264a26fb]). Expected version 1 got 0
I20240502 20:46:10.067122 4235 catalog_manager.cc:5576] TS ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193) does not have the latest schema for tablet 1fc0c640a6fb4b24b17642ed26ce8f07 (table rebalancer_dynamic_table [id=44b9d8eb907c483bb3ba9399264a26fb]). Expected version 1 got 0
I20240502 20:46:10.067471 4230 catalog_manager.cc:5484] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index 5 to 6, VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) evicted. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:10.069154 6967 raft_consensus.cc:2932] T 0cf7e95cb91b4e74b14365fcc6b7f968 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:10.069228 4768 raft_consensus.cc:2932] T 0cf7e95cb91b4e74b14365fcc6b7f968 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:10.071632 4356 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cf7e95cb91b4e74b14365fcc6b7f968 with delete_type TABLET_DATA_TOMBSTONED (TS c92e6e46782849479c8a0c6d82276fb2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38306
I20240502 20:46:10.071867 7297 tablet_replica.cc:330] T 0cf7e95cb91b4e74b14365fcc6b7f968 P c92e6e46782849479c8a0c6d82276fb2: stopping tablet replica
I20240502 20:46:10.072006 7297 raft_consensus.cc:2229] T 0cf7e95cb91b4e74b14365fcc6b7f968 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:10.072149 7297 raft_consensus.cc:2258] T 0cf7e95cb91b4e74b14365fcc6b7f968 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:10.073021 7297 ts_tablet_manager.cc:1875] T 0cf7e95cb91b4e74b14365fcc6b7f968 P c92e6e46782849479c8a0c6d82276fb2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:10.073792 8111 raft_consensus.cc:1050] T a55d5b8b9ed04ca39888b89b82a3f833 P ed47c023452a497192f35d2f617bf9bd: attempting to promote NON_VOTER 7845c397230d49358ba2fad9db53f97c to VOTER
I20240502 20:46:10.074216 8111 consensus_queue.cc:237] T a55d5b8b9ed04ca39888b89b82a3f833 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } }
I20240502 20:46:10.075629 7297 ts_tablet_manager.cc:1888] T 0cf7e95cb91b4e74b14365fcc6b7f968 P c92e6e46782849479c8a0c6d82276fb2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:46:10.075711 7297 log.cc:1196] T 0cf7e95cb91b4e74b14365fcc6b7f968 P c92e6e46782849479c8a0c6d82276fb2: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal/wals/0cf7e95cb91b4e74b14365fcc6b7f968
I20240502 20:46:10.075762 4374 raft_consensus.cc:1261] T a55d5b8b9ed04ca39888b89b82a3f833 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:10.076153 6241 consensus_queue.cc:1035] T a55d5b8b9ed04ca39888b89b82a3f833 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:10.076299 4222 catalog_manager.cc:4830] TS c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827): tablet 0cf7e95cb91b4e74b14365fcc6b7f968 (table rebalance_test_table_17 [id=75dfa602f9634de99d6f55e4fdeea21c]) successfully deleted
I20240502 20:46:10.076501 4507 raft_consensus.cc:1261] T a55d5b8b9ed04ca39888b89b82a3f833 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:10.076589 6703 raft_consensus.cc:1261] T a55d5b8b9ed04ca39888b89b82a3f833 P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:10.077118 6375 raft_consensus.cc:1050] T 5053a7afb4994ecdbc92b129c58903cf P c3d26eafd51547acb780d4aac5ebe7cc: attempting to promote NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 to VOTER
I20240502 20:46:10.077560 6375 consensus_queue.cc:237] T 5053a7afb4994ecdbc92b129c58903cf P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } }
I20240502 20:46:10.079386 4638 raft_consensus.cc:1261] T 5053a7afb4994ecdbc92b129c58903cf P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:10.079706 6241 consensus_queue.cc:1035] T a55d5b8b9ed04ca39888b89b82a3f833 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:10.079775 6835 raft_consensus.cc:1261] T 5053a7afb4994ecdbc92b129c58903cf P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEARNER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:10.079905 7967 consensus_queue.cc:1035] T 5053a7afb4994ecdbc92b129c58903cf P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:10.079998 9069 consensus_queue.cc:1035] T a55d5b8b9ed04ca39888b89b82a3f833 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:10.080103 4768 raft_consensus.cc:1261] T 5053a7afb4994ecdbc92b129c58903cf P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:10.080523 7967 consensus_queue.cc:1035] T 5053a7afb4994ecdbc92b129c58903cf P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:10.080857 8146 consensus_queue.cc:1035] T 5053a7afb4994ecdbc92b129c58903cf P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:10.083941 8907 raft_consensus.cc:2932] T 5053a7afb4994ecdbc92b129c58903cf P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:10.084311 8111 raft_consensus.cc:2932] T a55d5b8b9ed04ca39888b89b82a3f833 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:10.085922 9113 tablet.cc:1722] T 1fc0c640a6fb4b24b17642ed26ce8f07 P a47e5c56f13843018ac584914a8d4707: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 20:46:10.086365 9113 tablet.cc:1620] T 1fc0c640a6fb4b24b17642ed26ce8f07 P a47e5c56f13843018ac584914a8d4707: MemRowSet was empty: no flush needed.
I20240502 20:46:10.086563 4374 raft_consensus.cc:2932] T a55d5b8b9ed04ca39888b89b82a3f833 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:10.087277 6835 raft_consensus.cc:2932] T 5053a7afb4994ecdbc92b129c58903cf P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:10.087985 4230 catalog_manager.cc:5484] T a55d5b8b9ed04ca39888b89b82a3f833 P ed47c023452a497192f35d2f617bf9bd reported cstate change: config changed from index 4 to 5, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:10.089618 4638 raft_consensus.cc:2932] T 5053a7afb4994ecdbc92b129c58903cf P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:10.089707 4228 catalog_manager.cc:5484] T 5053a7afb4994ecdbc92b129c58903cf P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: config changed from index 4 to 5, a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:10.091293 4768 raft_consensus.cc:2932] T 5053a7afb4994ecdbc92b129c58903cf P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:10.091684 6703 raft_consensus.cc:2932] T a55d5b8b9ed04ca39888b89b82a3f833 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:10.092706 6835 raft_consensus.cc:1203] T dbc1ef85e30d438682bc86935ceefd1b P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:10.095222 4507 raft_consensus.cc:2932] T a55d5b8b9ed04ca39888b89b82a3f833 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:10.097311 4235 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: rebalancer_dynamic_table [id=44b9d8eb907c483bb3ba9399264a26fb] alter complete (version 1)
I20240502 20:46:10.099871 9114 tablet.cc:1722] T 1fc0c640a6fb4b24b17642ed26ce8f07 P 201d2b6485084e7dad2cbb722af94779: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 20:46:10.100584 4507 consensus_queue.cc:237] T 5053a7afb4994ecdbc92b129c58903cf P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } }
I20240502 20:46:10.101588 9114 tablet.cc:1620] T 1fc0c640a6fb4b24b17642ed26ce8f07 P 201d2b6485084e7dad2cbb722af94779: MemRowSet was empty: no flush needed.
I20240502 20:46:10.102834 4768 consensus_queue.cc:237] T a55d5b8b9ed04ca39888b89b82a3f833 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } }
I20240502 20:46:10.104187 4374 raft_consensus.cc:1261] T a55d5b8b9ed04ca39888b89b82a3f833 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:10.104373 6835 raft_consensus.cc:1261] T 5053a7afb4994ecdbc92b129c58903cf P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:10.104842 7996 consensus_queue.cc:1035] T a55d5b8b9ed04ca39888b89b82a3f833 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:10.104940 8907 consensus_queue.cc:1035] T 5053a7afb4994ecdbc92b129c58903cf P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:10.105578 9126 ts_tablet_manager.cc:902] T 15a2cbe8285a4938be7f628f7f8b734d P 923cfab926e7443f8ffd7b21ad0a9d91: Initiating tablet copy from peer c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:46:10.106431 9126 tablet_copy_client.cc:308] T 15a2cbe8285a4938be7f628f7f8b734d P 923cfab926e7443f8ffd7b21ad0a9d91: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.65:33827
I20240502 20:46:10.106760 7996 raft_consensus.cc:2932] T a55d5b8b9ed04ca39888b89b82a3f833 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:10.107138 4397 tablet_copy_service.cc:140] P c92e6e46782849479c8a0c6d82276fb2: Received BeginTabletCopySession request for tablet 15a2cbe8285a4938be7f628f7f8b734d from peer 923cfab926e7443f8ffd7b21ad0a9d91 ({username='slave'} at 127.30.32.67:51233)
I20240502 20:46:10.107234 4397 tablet_copy_service.cc:161] P c92e6e46782849479c8a0c6d82276fb2: Beginning new tablet copy session on tablet 15a2cbe8285a4938be7f628f7f8b734d from peer 923cfab926e7443f8ffd7b21ad0a9d91 at {username='slave'} at 127.30.32.67:51233: session id = 923cfab926e7443f8ffd7b21ad0a9d91-15a2cbe8285a4938be7f628f7f8b734d
I20240502 20:46:10.107947 4397 tablet_copy_source_session.cc:205] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:10.108140 4222 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a55d5b8b9ed04ca39888b89b82a3f833 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:10.108562 4374 raft_consensus.cc:2932] T a55d5b8b9ed04ca39888b89b82a3f833 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:10.109181 9126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15a2cbe8285a4938be7f628f7f8b734d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:10.110069 4228 catalog_manager.cc:5484] T a55d5b8b9ed04ca39888b89b82a3f833 P ed47c023452a497192f35d2f617bf9bd reported cstate change: config changed from index 5 to 6, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:10.111963 6703 raft_consensus.cc:1261] T a55d5b8b9ed04ca39888b89b82a3f833 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:10.113039 4488 tablet_service.cc:1514] Processing DeleteTablet for tablet a55d5b8b9ed04ca39888b89b82a3f833 with delete_type TABLET_DATA_TOMBSTONED (TS c3d26eafd51547acb780d4aac5ebe7cc not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48890
I20240502 20:46:10.113240 8141 tablet_replica.cc:330] T a55d5b8b9ed04ca39888b89b82a3f833 P c3d26eafd51547acb780d4aac5ebe7cc: stopping tablet replica
I20240502 20:46:10.113373 8141 raft_consensus.cc:2229] T a55d5b8b9ed04ca39888b89b82a3f833 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:10.113529 8141 raft_consensus.cc:2258] T a55d5b8b9ed04ca39888b89b82a3f833 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:10.113652 9069 consensus_queue.cc:1035] T a55d5b8b9ed04ca39888b89b82a3f833 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:10.113654 4769 raft_consensus.cc:1261] T 5053a7afb4994ecdbc92b129c58903cf P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:10.114322 8141 ts_tablet_manager.cc:1875] T a55d5b8b9ed04ca39888b89b82a3f833 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:10.114768 6703 raft_consensus.cc:2932] T a55d5b8b9ed04ca39888b89b82a3f833 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:10.116025 8907 consensus_queue.cc:1035] T 5053a7afb4994ecdbc92b129c58903cf P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:10.116492 8141 ts_tablet_manager.cc:1888] T a55d5b8b9ed04ca39888b89b82a3f833 P c3d26eafd51547acb780d4aac5ebe7cc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:46:10.116562 8141 log.cc:1196] T a55d5b8b9ed04ca39888b89b82a3f833 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal/wals/a55d5b8b9ed04ca39888b89b82a3f833
I20240502 20:46:10.116640 8907 raft_consensus.cc:2932] T 5053a7afb4994ecdbc92b129c58903cf P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:10.118453 4221 catalog_manager.cc:4830] TS c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097): tablet a55d5b8b9ed04ca39888b89b82a3f833 (table rebalance_test_table_18 [id=393c8bbf6b174a9e891d96bcd09fe045]) successfully deleted
I20240502 20:46:10.118629 4221 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5053a7afb4994ecdbc92b129c58903cf with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:10.117956 9126 tablet_copy_client.cc:783] T 15a2cbe8285a4938be7f628f7f8b734d P 923cfab926e7443f8ffd7b21ad0a9d91: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:10.118968 9126 tablet_copy_client.cc:647] T 15a2cbe8285a4938be7f628f7f8b734d P 923cfab926e7443f8ffd7b21ad0a9d91: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:10.119431 4231 catalog_manager.cc:5484] T 5053a7afb4994ecdbc92b129c58903cf P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: config changed from index 5 to 6, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:10.119439 6835 raft_consensus.cc:2932] T 5053a7afb4994ecdbc92b129c58903cf P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:10.120543 9126 tablet_copy_client.cc:523] T 15a2cbe8285a4938be7f628f7f8b734d P 923cfab926e7443f8ffd7b21ad0a9d91: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:10.121217 4768 raft_consensus.cc:2932] T 5053a7afb4994ecdbc92b129c58903cf P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:10.122334 4618 tablet_service.cc:1514] Processing DeleteTablet for tablet 5053a7afb4994ecdbc92b129c58903cf with delete_type TABLET_DATA_TOMBSTONED (TS 923cfab926e7443f8ffd7b21ad0a9d91 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57068
I20240502 20:46:10.122566 8510 tablet_replica.cc:330] T 5053a7afb4994ecdbc92b129c58903cf P 923cfab926e7443f8ffd7b21ad0a9d91: stopping tablet replica
I20240502 20:46:10.122711 8510 raft_consensus.cc:2229] T 5053a7afb4994ecdbc92b129c58903cf P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:10.122881 8510 raft_consensus.cc:2258] T 5053a7afb4994ecdbc92b129c58903cf P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:10.123579 9126 tablet_bootstrap.cc:492] T 15a2cbe8285a4938be7f628f7f8b734d P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:46:10.123670 8510 ts_tablet_manager.cc:1875] T 5053a7afb4994ecdbc92b129c58903cf P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:10.127027 8510 ts_tablet_manager.cc:1888] T 5053a7afb4994ecdbc92b129c58903cf P 923cfab926e7443f8ffd7b21ad0a9d91: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:46:10.127138 8510 log.cc:1196] T 5053a7afb4994ecdbc92b129c58903cf P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal/wals/5053a7afb4994ecdbc92b129c58903cf
I20240502 20:46:10.127826 9126 tablet_bootstrap.cc:492] T 15a2cbe8285a4938be7f628f7f8b734d P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:10.128015 4219 catalog_manager.cc:4830] TS 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141): tablet 5053a7afb4994ecdbc92b129c58903cf (table rebalance_test_table_5 [id=5c60c96389ff4cf3bbd5edb6a694c70d]) successfully deleted
I20240502 20:46:10.128101 9126 tablet_bootstrap.cc:492] T 15a2cbe8285a4938be7f628f7f8b734d P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap complete.
I20240502 20:46:10.128247 9126 ts_tablet_manager.cc:1367] T 15a2cbe8285a4938be7f628f7f8b734d P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 20:46:10.128247 9131 tablet.cc:1722] T 1fc0c640a6fb4b24b17642ed26ce8f07 P ed47c023452a497192f35d2f617bf9bd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 20:46:10.128597 9131 tablet.cc:1620] T 1fc0c640a6fb4b24b17642ed26ce8f07 P ed47c023452a497192f35d2f617bf9bd: MemRowSet was empty: no flush needed.
I20240502 20:46:10.128655 9126 raft_consensus.cc:348] T 15a2cbe8285a4938be7f628f7f8b734d P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } }
I20240502 20:46:10.128767 9126 raft_consensus.cc:729] T 15a2cbe8285a4938be7f628f7f8b734d P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: LEARNER
I20240502 20:46:10.128854 9126 consensus_queue.cc:260] T 15a2cbe8285a4938be7f628f7f8b734d P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } }
I20240502 20:46:10.129137 9126 ts_tablet_manager.cc:1398] T 15a2cbe8285a4938be7f628f7f8b734d P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:10.129891 4397 tablet_copy_service.cc:342] P c92e6e46782849479c8a0c6d82276fb2: Request end of tablet copy session 923cfab926e7443f8ffd7b21ad0a9d91-15a2cbe8285a4938be7f628f7f8b734d received from {username='slave'} at 127.30.32.67:51233
I20240502 20:46:10.129983 4397 tablet_copy_service.cc:433] P c92e6e46782849479c8a0c6d82276fb2: ending tablet copy session 923cfab926e7443f8ffd7b21ad0a9d91-15a2cbe8285a4938be7f628f7f8b734d on tablet 15a2cbe8285a4938be7f628f7f8b734d with peer 923cfab926e7443f8ffd7b21ad0a9d91
I20240502 20:46:10.142879 9105 raft_consensus.cc:482] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:10.143015 9105 raft_consensus.cc:504] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:10.143417 9105 leader_election.cc:288] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867)
I20240502 20:46:10.143776 6967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9afb75bfc114f59850eaea084a306f0" candidate_uuid: "7845c397230d49358ba2fad9db53f97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47e5c56f13843018ac584914a8d4707" is_pre_election: true
I20240502 20:46:10.143828 6835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9afb75bfc114f59850eaea084a306f0" candidate_uuid: "7845c397230d49358ba2fad9db53f97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833" is_pre_election: true
I20240502 20:46:10.143946 6967 raft_consensus.cc:2454] T a9afb75bfc114f59850eaea084a306f0 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7845c397230d49358ba2fad9db53f97c in term 0.
I20240502 20:46:10.143949 6835 raft_consensus.cc:2454] T a9afb75bfc114f59850eaea084a306f0 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7845c397230d49358ba2fad9db53f97c in term 0.
I20240502 20:46:10.144248 6637 leader_election.cc:302] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7845c397230d49358ba2fad9db53f97c, a4bf1df7fa6b4948be502c182bfbb833; no voters:
I20240502 20:46:10.144438 9105 raft_consensus.cc:2781] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:10.144502 9105 raft_consensus.cc:482] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:10.144528 9105 raft_consensus.cc:3037] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:10.145404 9105 raft_consensus.cc:504] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:10.145727 9105 leader_election.cc:288] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 election: Requested vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867)
I20240502 20:46:10.146093 6967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9afb75bfc114f59850eaea084a306f0" candidate_uuid: "7845c397230d49358ba2fad9db53f97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47e5c56f13843018ac584914a8d4707"
I20240502 20:46:10.146093 6835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9afb75bfc114f59850eaea084a306f0" candidate_uuid: "7845c397230d49358ba2fad9db53f97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833"
I20240502 20:46:10.146216 6835 raft_consensus.cc:3037] T a9afb75bfc114f59850eaea084a306f0 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:10.146216 6967 raft_consensus.cc:3037] T a9afb75bfc114f59850eaea084a306f0 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:10.147171 6967 raft_consensus.cc:2454] T a9afb75bfc114f59850eaea084a306f0 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7845c397230d49358ba2fad9db53f97c in term 1.
I20240502 20:46:10.147171 6835 raft_consensus.cc:2454] T a9afb75bfc114f59850eaea084a306f0 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7845c397230d49358ba2fad9db53f97c in term 1.
I20240502 20:46:10.147512 6639 leader_election.cc:302] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7845c397230d49358ba2fad9db53f97c, a47e5c56f13843018ac584914a8d4707; no voters:
I20240502 20:46:10.147747 9105 raft_consensus.cc:2781] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:10.147851 9105 raft_consensus.cc:686] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c [term 1 LEADER]: Becoming Leader. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Running, Role: LEADER
I20240502 20:46:10.148018 9105 consensus_queue.cc:237] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:10.149946 4231 catalog_manager.cc:5484] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7845c397230d49358ba2fad9db53f97c (127.30.32.69). New cstate: current_term: 1 leader_uuid: "7845c397230d49358ba2fad9db53f97c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:10.153033 6967 raft_consensus.cc:1203] T 7abfd61f55554042978b50761cbd6ab5 P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 20:46:10.204389 4228 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 20:46:10.210019 9113 tablet.cc:1722] T 1fc0c640a6fb4b24b17642ed26ce8f07 P a47e5c56f13843018ac584914a8d4707: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 20:46:10.210363 9113 tablet.cc:1620] T 1fc0c640a6fb4b24b17642ed26ce8f07 P a47e5c56f13843018ac584914a8d4707: MemRowSet was empty: no flush needed.
I20240502 20:46:10.210721 9114 tablet.cc:1722] T 1fc0c640a6fb4b24b17642ed26ce8f07 P 201d2b6485084e7dad2cbb722af94779: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 20:46:10.211002 9114 tablet.cc:1620] T 1fc0c640a6fb4b24b17642ed26ce8f07 P 201d2b6485084e7dad2cbb722af94779: MemRowSet was empty: no flush needed.
I20240502 20:46:10.211594 4231 catalog_manager.cc:5576] TS a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841) does not have the latest schema for tablet 1fc0c640a6fb4b24b17642ed26ce8f07 (table rebalancer_dynamic_table [id=44b9d8eb907c483bb3ba9399264a26fb]). Expected version 2 got 1
I20240502 20:46:10.212263 9131 tablet.cc:1722] T 1fc0c640a6fb4b24b17642ed26ce8f07 P ed47c023452a497192f35d2f617bf9bd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 20:46:10.212535 9131 tablet.cc:1620] T 1fc0c640a6fb4b24b17642ed26ce8f07 P ed47c023452a497192f35d2f617bf9bd: MemRowSet was empty: no flush needed.
I20240502 20:46:10.218297 4231 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: rebalancer_dynamic_table [id=44b9d8eb907c483bb3ba9399264a26fb] alter complete (version 2)
I20240502 20:46:10.219655 9105 raft_consensus.cc:482] T 77dc51e6a85a4ddcb64db12b40c5810f P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:10.219767 9105 raft_consensus.cc:504] T 77dc51e6a85a4ddcb64db12b40c5810f P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:10.220188 9105 leader_election.cc:288] T 77dc51e6a85a4ddcb64db12b40c5810f P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141), c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097)
I20240502 20:46:10.220490 4638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77dc51e6a85a4ddcb64db12b40c5810f" candidate_uuid: "7845c397230d49358ba2fad9db53f97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" is_pre_election: true
I20240502 20:46:10.220516 4507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77dc51e6a85a4ddcb64db12b40c5810f" candidate_uuid: "7845c397230d49358ba2fad9db53f97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" is_pre_election: true
I20240502 20:46:10.220626 4638 raft_consensus.cc:2454] T 77dc51e6a85a4ddcb64db12b40c5810f P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7845c397230d49358ba2fad9db53f97c in term 0.
I20240502 20:46:10.220643 4507 raft_consensus.cc:2454] T 77dc51e6a85a4ddcb64db12b40c5810f P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7845c397230d49358ba2fad9db53f97c in term 0.
I20240502 20:46:10.220904 6636 leader_election.cc:302] T 77dc51e6a85a4ddcb64db12b40c5810f P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7845c397230d49358ba2fad9db53f97c, 923cfab926e7443f8ffd7b21ad0a9d91; no voters:
I20240502 20:46:10.221177 9105 raft_consensus.cc:2781] T 77dc51e6a85a4ddcb64db12b40c5810f P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:10.221221 9105 raft_consensus.cc:482] T 77dc51e6a85a4ddcb64db12b40c5810f P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:10.221256 9105 raft_consensus.cc:3037] T 77dc51e6a85a4ddcb64db12b40c5810f P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:10.222265 9105 raft_consensus.cc:504] T 77dc51e6a85a4ddcb64db12b40c5810f P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:10.222602 9105 leader_election.cc:288] T 77dc51e6a85a4ddcb64db12b40c5810f P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 election: Requested vote from peers 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141), c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097)
I20240502 20:46:10.223012 4507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77dc51e6a85a4ddcb64db12b40c5810f" candidate_uuid: "7845c397230d49358ba2fad9db53f97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc"
I20240502 20:46:10.223002 4638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77dc51e6a85a4ddcb64db12b40c5810f" candidate_uuid: "7845c397230d49358ba2fad9db53f97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91"
I20240502 20:46:10.223122 4638 raft_consensus.cc:3037] T 77dc51e6a85a4ddcb64db12b40c5810f P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:10.223122 4507 raft_consensus.cc:3037] T 77dc51e6a85a4ddcb64db12b40c5810f P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:10.224092 4638 raft_consensus.cc:2454] T 77dc51e6a85a4ddcb64db12b40c5810f P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7845c397230d49358ba2fad9db53f97c in term 1.
I20240502 20:46:10.224092 4507 raft_consensus.cc:2454] T 77dc51e6a85a4ddcb64db12b40c5810f P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7845c397230d49358ba2fad9db53f97c in term 1.
I20240502 20:46:10.224431 6636 leader_election.cc:302] T 77dc51e6a85a4ddcb64db12b40c5810f P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7845c397230d49358ba2fad9db53f97c, 923cfab926e7443f8ffd7b21ad0a9d91; no voters:
I20240502 20:46:10.224599 9105 raft_consensus.cc:2781] T 77dc51e6a85a4ddcb64db12b40c5810f P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:10.224694 9105 raft_consensus.cc:686] T 77dc51e6a85a4ddcb64db12b40c5810f P 7845c397230d49358ba2fad9db53f97c [term 1 LEADER]: Becoming Leader. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Running, Role: LEADER
I20240502 20:46:10.224828 9105 consensus_queue.cc:237] T 77dc51e6a85a4ddcb64db12b40c5810f P 7845c397230d49358ba2fad9db53f97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:10.226866 4231 catalog_manager.cc:5484] T 77dc51e6a85a4ddcb64db12b40c5810f P 7845c397230d49358ba2fad9db53f97c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7845c397230d49358ba2fad9db53f97c (127.30.32.69). New cstate: current_term: 1 leader_uuid: "7845c397230d49358ba2fad9db53f97c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:10.248122 9148 ts_tablet_manager.cc:902] T d6fb394fdcff436886fe0634d79c024c P 201d2b6485084e7dad2cbb722af94779: Initiating tablet copy from peer c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097)
I20240502 20:46:10.248656 9148 tablet_copy_client.cc:308] T d6fb394fdcff436886fe0634d79c024c P 201d2b6485084e7dad2cbb722af94779: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.66:42097
I20240502 20:46:10.249441 4528 tablet_copy_service.cc:140] P c3d26eafd51547acb780d4aac5ebe7cc: Received BeginTabletCopySession request for tablet d6fb394fdcff436886fe0634d79c024c from peer 201d2b6485084e7dad2cbb722af94779 ({username='slave'} at 127.30.32.72:54159)
I20240502 20:46:10.249540 4528 tablet_copy_service.cc:161] P c3d26eafd51547acb780d4aac5ebe7cc: Beginning new tablet copy session on tablet d6fb394fdcff436886fe0634d79c024c from peer 201d2b6485084e7dad2cbb722af94779 at {username='slave'} at 127.30.32.72:54159: session id = 201d2b6485084e7dad2cbb722af94779-d6fb394fdcff436886fe0634d79c024c
I20240502 20:46:10.250455 4528 tablet_copy_source_session.cc:205] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:10.253953 9148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6fb394fdcff436886fe0634d79c024c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:10.256908 7079 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:10.259742 6947 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:46:10.260383 4618 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 20:46:10.260852 9148 tablet_copy_client.cc:783] T d6fb394fdcff436886fe0634d79c024c P 201d2b6485084e7dad2cbb722af94779: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:10.261485 9148 tablet_copy_client.cc:647] T d6fb394fdcff436886fe0634d79c024c P 201d2b6485084e7dad2cbb722af94779: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:10.262817 9148 tablet_copy_client.cc:523] T d6fb394fdcff436886fe0634d79c024c P 201d2b6485084e7dad2cbb722af94779: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:10.264195 9148 tablet_bootstrap.cc:492] T d6fb394fdcff436886fe0634d79c024c P 201d2b6485084e7dad2cbb722af94779: Bootstrap starting.
I20240502 20:46:10.274911 6815 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:46:10.277842 4356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:10.290771 4488 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:10.293336 4750 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:10.303422 9148 tablet_bootstrap.cc:492] T d6fb394fdcff436886fe0634d79c024c P 201d2b6485084e7dad2cbb722af94779: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:10.303746 9148 tablet_bootstrap.cc:492] T d6fb394fdcff436886fe0634d79c024c P 201d2b6485084e7dad2cbb722af94779: Bootstrap complete.
I20240502 20:46:10.303931 9148 ts_tablet_manager.cc:1367] T d6fb394fdcff436886fe0634d79c024c P 201d2b6485084e7dad2cbb722af94779: Time spent bootstrapping tablet: real 0.040s user 0.025s sys 0.000s
I20240502 20:46:10.304389 9148 raft_consensus.cc:348] T d6fb394fdcff436886fe0634d79c024c P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } }
I20240502 20:46:10.304525 9148 raft_consensus.cc:729] T d6fb394fdcff436886fe0634d79c024c P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Initialized, Role: LEARNER
I20240502 20:46:10.304694 9148 consensus_queue.cc:260] T d6fb394fdcff436886fe0634d79c024c P 201d2b6485084e7dad2cbb722af94779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } }
I20240502 20:46:10.305130 9148 ts_tablet_manager.cc:1398] T d6fb394fdcff436886fe0634d79c024c P 201d2b6485084e7dad2cbb722af94779: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240502 20:46:10.306030 4528 tablet_copy_service.cc:342] P c3d26eafd51547acb780d4aac5ebe7cc: Request end of tablet copy session 201d2b6485084e7dad2cbb722af94779-d6fb394fdcff436886fe0634d79c024c received from {username='slave'} at 127.30.32.72:54159
I20240502 20:46:10.306116 4528 tablet_copy_service.cc:433] P c3d26eafd51547acb780d4aac5ebe7cc: ending tablet copy session 201d2b6485084e7dad2cbb722af94779-d6fb394fdcff436886fe0634d79c024c on tablet d6fb394fdcff436886fe0634d79c024c with peer 201d2b6485084e7dad2cbb722af94779
I20240502 20:46:10.309988 6681 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:10.320525 4231 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 20:46:10.323421 8304 raft_consensus.cc:1050] T 747e7331d7ad4dd4b132672254037e1b P a47e5c56f13843018ac584914a8d4707: attempting to promote NON_VOTER c3d26eafd51547acb780d4aac5ebe7cc to VOTER
I20240502 20:46:10.323927 8304 consensus_queue.cc:237] T 747e7331d7ad4dd4b132672254037e1b P a47e5c56f13843018ac584914a8d4707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } }
I20240502 20:46:10.325587 4507 raft_consensus.cc:1261] T 747e7331d7ad4dd4b132672254037e1b P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEARNER]: Refusing update from remote peer a47e5c56f13843018ac584914a8d4707: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:10.325767 6703 raft_consensus.cc:1261] T 747e7331d7ad4dd4b132672254037e1b P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Refusing update from remote peer a47e5c56f13843018ac584914a8d4707: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:10.325783 6835 raft_consensus.cc:1261] T 747e7331d7ad4dd4b132672254037e1b P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Refusing update from remote peer a47e5c56f13843018ac584914a8d4707: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:10.326301 8528 consensus_queue.cc:1035] T 747e7331d7ad4dd4b132672254037e1b P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:46:10.326634 8528 consensus_queue.cc:1035] T 747e7331d7ad4dd4b132672254037e1b P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:46:10.327234 8528 consensus_queue.cc:1035] T 747e7331d7ad4dd4b132672254037e1b P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:46:10.330166 9113 tablet.cc:1722] T 1fc0c640a6fb4b24b17642ed26ce8f07 P a47e5c56f13843018ac584914a8d4707: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 20:46:10.330716 9131 tablet.cc:1722] T 1fc0c640a6fb4b24b17642ed26ce8f07 P ed47c023452a497192f35d2f617bf9bd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 20:46:10.332764 9159 ts_tablet_manager.cc:902] T 8565ebbd21dd4f708a0254d1ed2d6670 P 7845c397230d49358ba2fad9db53f97c: Initiating tablet copy from peer 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:46:10.333474 9159 tablet_copy_client.cc:308] T 8565ebbd21dd4f708a0254d1ed2d6670 P 7845c397230d49358ba2fad9db53f97c: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.67:42141
I20240502 20:46:10.335283 4658 tablet_copy_service.cc:140] P 923cfab926e7443f8ffd7b21ad0a9d91: Received BeginTabletCopySession request for tablet 8565ebbd21dd4f708a0254d1ed2d6670 from peer 7845c397230d49358ba2fad9db53f97c ({username='slave'} at 127.30.32.69:34201)
I20240502 20:46:10.335394 4658 tablet_copy_service.cc:161] P 923cfab926e7443f8ffd7b21ad0a9d91: Beginning new tablet copy session on tablet 8565ebbd21dd4f708a0254d1ed2d6670 from peer 7845c397230d49358ba2fad9db53f97c at {username='slave'} at 127.30.32.69:34201: session id = 7845c397230d49358ba2fad9db53f97c-8565ebbd21dd4f708a0254d1ed2d6670
I20240502 20:46:10.335875 4228 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: rebalancer_dynamic_table_new_name [id=44b9d8eb907c483bb3ba9399264a26fb] alter complete (version 3)
I20240502 20:46:10.336433 4658 tablet_copy_source_session.cc:205] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:10.337061 9114 tablet.cc:1722] T 1fc0c640a6fb4b24b17642ed26ce8f07 P 201d2b6485084e7dad2cbb722af94779: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 20:46:10.337162 6138 raft_consensus.cc:1050] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2: attempting to promote NON_VOTER a47e5c56f13843018ac584914a8d4707 to VOTER
I20240502 20:46:10.337363 9159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8565ebbd21dd4f708a0254d1ed2d6670. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:10.337563 6138 consensus_queue.cc:237] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } }
I20240502 20:46:10.337846 4233 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240502 20:46:10.337950 4233 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240502 20:46:10.339286 6967 raft_consensus.cc:1261] T c55ba788f592448d8941ff94a8cefc52 P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:10.339308 4507 raft_consensus.cc:1261] T c55ba788f592448d8941ff94a8cefc52 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:10.339666 8918 consensus_queue.cc:1035] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:10.339789 4638 raft_consensus.cc:1261] T c55ba788f592448d8941ff94a8cefc52 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:10.340065 8185 consensus_queue.cc:1035] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:10.340263 7937 consensus_queue.cc:1035] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:10.340305 9159 tablet_copy_client.cc:783] T 8565ebbd21dd4f708a0254d1ed2d6670 P 7845c397230d49358ba2fad9db53f97c: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:10.340447 9159 tablet_copy_client.cc:647] T 8565ebbd21dd4f708a0254d1ed2d6670 P 7845c397230d49358ba2fad9db53f97c: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:10.341818 4233 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: Sending DeleteTablet for 3 replicas of tablet 66b8f583dfd24c2aa5a59457a8e58fbb
I20240502 20:46:10.342001 9159 tablet_copy_client.cc:523] T 8565ebbd21dd4f708a0254d1ed2d6670 P 7845c397230d49358ba2fad9db53f97c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:10.342123 4233 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: Sending DeleteTablet for 3 replicas of tablet c6ac59f330454d2a89ad9c11192ed4f0
I20240502 20:46:10.342384 4233 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: Sending DeleteTablet for 3 replicas of tablet a55d5b8b9ed04ca39888b89b82a3f833
I20240502 20:46:10.343461 9159 tablet_bootstrap.cc:492] T 8565ebbd21dd4f708a0254d1ed2d6670 P 7845c397230d49358ba2fad9db53f97c: Bootstrap starting.
I20240502 20:46:10.343796 4488 tablet_service.cc:1514] Processing DeleteTablet for tablet 66b8f583dfd24c2aa5a59457a8e58fbb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:10 UTC) from {username='slave'} at 127.0.0.1:48890
I20240502 20:46:10.343920 6681 tablet_service.cc:1514] Processing DeleteTablet for tablet a55d5b8b9ed04ca39888b89b82a3f833 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:10 UTC) from {username='slave'} at 127.0.0.1:37506
I20240502 20:46:10.343958 4487 tablet_service.cc:1514] Processing DeleteTablet for tablet c6ac59f330454d2a89ad9c11192ed4f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:10 UTC) from {username='slave'} at 127.0.0.1:48890
I20240502 20:46:10.344100 8141 tablet_replica.cc:330] T 66b8f583dfd24c2aa5a59457a8e58fbb P c3d26eafd51547acb780d4aac5ebe7cc: stopping tablet replica
I20240502 20:46:10.344172 4618 tablet_service.cc:1514] Processing DeleteTablet for tablet c6ac59f330454d2a89ad9c11192ed4f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:10 UTC) from {username='slave'} at 127.0.0.1:57068
I20240502 20:46:10.344245 8141 raft_consensus.cc:2229] T 66b8f583dfd24c2aa5a59457a8e58fbb P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:10.344342 8510 tablet_replica.cc:330] T c6ac59f330454d2a89ad9c11192ed4f0 P 923cfab926e7443f8ffd7b21ad0a9d91: stopping tablet replica
I20240502 20:46:10.344450 8141 raft_consensus.cc:2258] T 66b8f583dfd24c2aa5a59457a8e58fbb P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:10.344473 8510 raft_consensus.cc:2229] T c6ac59f330454d2a89ad9c11192ed4f0 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:10.344578 4749 tablet_service.cc:1514] Processing DeleteTablet for tablet a55d5b8b9ed04ca39888b89b82a3f833 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:10 UTC) from {username='slave'} at 127.0.0.1:38526
I20240502 20:46:10.344624 8510 raft_consensus.cc:2258] T c6ac59f330454d2a89ad9c11192ed4f0 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:10.344802 4356 tablet_service.cc:1514] Processing DeleteTablet for tablet 66b8f583dfd24c2aa5a59457a8e58fbb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:10 UTC) from {username='slave'} at 127.0.0.1:38306
I20240502 20:46:10.344578 4750 tablet_service.cc:1514] Processing DeleteTablet for tablet c6ac59f330454d2a89ad9c11192ed4f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:10 UTC) from {username='slave'} at 127.0.0.1:38526
I20240502 20:46:10.345121 7299 tablet_replica.cc:330] T a55d5b8b9ed04ca39888b89b82a3f833 P ed47c023452a497192f35d2f617bf9bd: stopping tablet replica
I20240502 20:46:10.345177 6947 tablet_service.cc:1514] Processing DeleteTablet for tablet 66b8f583dfd24c2aa5a59457a8e58fbb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:10 UTC) from {username='slave'} at 127.0.0.1:45156
I20240502 20:46:10.345211 8141 ts_tablet_manager.cc:1875] T 66b8f583dfd24c2aa5a59457a8e58fbb P c3d26eafd51547acb780d4aac5ebe7cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:10.345243 7299 raft_consensus.cc:2229] T a55d5b8b9ed04ca39888b89b82a3f833 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:10.345472 7299 raft_consensus.cc:2258] T a55d5b8b9ed04ca39888b89b82a3f833 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:10.346335 7299 ts_tablet_manager.cc:1875] T a55d5b8b9ed04ca39888b89b82a3f833 P ed47c023452a497192f35d2f617bf9bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:10.347386 8141 ts_tablet_manager.cc:1888] T 66b8f583dfd24c2aa5a59457a8e58fbb P c3d26eafd51547acb780d4aac5ebe7cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240502 20:46:10.347445 8141 log.cc:1196] T 66b8f583dfd24c2aa5a59457a8e58fbb P c3d26eafd51547acb780d4aac5ebe7cc: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal/wals/66b8f583dfd24c2aa5a59457a8e58fbb
I20240502 20:46:10.347677 8141 ts_tablet_manager.cc:1909] T 66b8f583dfd24c2aa5a59457a8e58fbb P c3d26eafd51547acb780d4aac5ebe7cc: Deleting consensus metadata
I20240502 20:46:10.348244 8141 tablet_replica.cc:330] T c6ac59f330454d2a89ad9c11192ed4f0 P c3d26eafd51547acb780d4aac5ebe7cc: stopping tablet replica
I20240502 20:46:10.348371 8141 raft_consensus.cc:2229] T c6ac59f330454d2a89ad9c11192ed4f0 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:10.348398 4221 catalog_manager.cc:4830] TS c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097): tablet 66b8f583dfd24c2aa5a59457a8e58fbb (table rebalance_test_table_18 [id=393c8bbf6b174a9e891d96bcd09fe045]) successfully deleted
I20240502 20:46:10.348426 7299 ts_tablet_manager.cc:1888] T a55d5b8b9ed04ca39888b89b82a3f833 P ed47c023452a497192f35d2f617bf9bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240502 20:46:10.348491 7299 log.cc:1196] T a55d5b8b9ed04ca39888b89b82a3f833 P ed47c023452a497192f35d2f617bf9bd: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal/wals/a55d5b8b9ed04ca39888b89b82a3f833
I20240502 20:46:10.348491 8141 raft_consensus.cc:2258] T c6ac59f330454d2a89ad9c11192ed4f0 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:10.348765 7299 ts_tablet_manager.cc:1909] T a55d5b8b9ed04ca39888b89b82a3f833 P ed47c023452a497192f35d2f617bf9bd: Deleting consensus metadata
I20240502 20:46:10.349103 4354 tablet_service.cc:1514] Processing DeleteTablet for tablet a55d5b8b9ed04ca39888b89b82a3f833 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:10 UTC) from {username='slave'} at 127.0.0.1:38306
I20240502 20:46:10.349282 7297 tablet_replica.cc:330] T 66b8f583dfd24c2aa5a59457a8e58fbb P c92e6e46782849479c8a0c6d82276fb2: stopping tablet replica
I20240502 20:46:10.349426 4222 catalog_manager.cc:4830] TS ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193): tablet a55d5b8b9ed04ca39888b89b82a3f833 (table rebalance_test_table_18 [id=393c8bbf6b174a9e891d96bcd09fe045]) successfully deleted
I20240502 20:46:10.349449 7297 raft_consensus.cc:2229] T 66b8f583dfd24c2aa5a59457a8e58fbb P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:10.349642 7297 raft_consensus.cc:2258] T 66b8f583dfd24c2aa5a59457a8e58fbb P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:10.349767 8510 ts_tablet_manager.cc:1875] T c6ac59f330454d2a89ad9c11192ed4f0 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:10.349817 9035 tablet_replica.cc:330] T 66b8f583dfd24c2aa5a59457a8e58fbb P a47e5c56f13843018ac584914a8d4707: stopping tablet replica
I20240502 20:46:10.349949 9035 raft_consensus.cc:2229] T 66b8f583dfd24c2aa5a59457a8e58fbb P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:10.350075 9035 raft_consensus.cc:2258] T 66b8f583dfd24c2aa5a59457a8e58fbb P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:10.349951 7937 raft_consensus.cc:2932] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:10.351634 7297 ts_tablet_manager.cc:1875] T 66b8f583dfd24c2aa5a59457a8e58fbb P c92e6e46782849479c8a0c6d82276fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:10.351877 8510 ts_tablet_manager.cc:1888] T c6ac59f330454d2a89ad9c11192ed4f0 P 923cfab926e7443f8ffd7b21ad0a9d91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 20:46:10.351938 8510 log.cc:1196] T c6ac59f330454d2a89ad9c11192ed4f0 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal/wals/c6ac59f330454d2a89ad9c11192ed4f0
I20240502 20:46:10.352178 8510 ts_tablet_manager.cc:1909] T c6ac59f330454d2a89ad9c11192ed4f0 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting consensus metadata
I20240502 20:46:10.353052 4219 catalog_manager.cc:4830] TS 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141): tablet c6ac59f330454d2a89ad9c11192ed4f0 (table rebalance_test_table_18 [id=393c8bbf6b174a9e891d96bcd09fe045]) successfully deleted
I20240502 20:46:10.353395 9035 ts_tablet_manager.cc:1875] T 66b8f583dfd24c2aa5a59457a8e58fbb P a47e5c56f13843018ac584914a8d4707: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:10.353469 4638 raft_consensus.cc:2932] T c55ba788f592448d8941ff94a8cefc52 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:10.353890 7297 ts_tablet_manager.cc:1888] T 66b8f583dfd24c2aa5a59457a8e58fbb P c92e6e46782849479c8a0c6d82276fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240502 20:46:10.353951 7297 log.cc:1196] T 66b8f583dfd24c2aa5a59457a8e58fbb P c92e6e46782849479c8a0c6d82276fb2: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal/wals/66b8f583dfd24c2aa5a59457a8e58fbb
I20240502 20:46:10.354205 7297 ts_tablet_manager.cc:1909] T 66b8f583dfd24c2aa5a59457a8e58fbb P c92e6e46782849479c8a0c6d82276fb2: Deleting consensus metadata
I20240502 20:46:10.355429 9035 ts_tablet_manager.cc:1888] T 66b8f583dfd24c2aa5a59457a8e58fbb P a47e5c56f13843018ac584914a8d4707: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240502 20:46:10.355417 4222 catalog_manager.cc:4830] TS c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827): tablet 66b8f583dfd24c2aa5a59457a8e58fbb (table rebalance_test_table_18 [id=393c8bbf6b174a9e891d96bcd09fe045]) successfully deleted
I20240502 20:46:10.355492 9035 log.cc:1196] T 66b8f583dfd24c2aa5a59457a8e58fbb P a47e5c56f13843018ac584914a8d4707: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/wal/wals/66b8f583dfd24c2aa5a59457a8e58fbb
I20240502 20:46:10.355752 9035 ts_tablet_manager.cc:1909] T 66b8f583dfd24c2aa5a59457a8e58fbb P a47e5c56f13843018ac584914a8d4707: Deleting consensus metadata
I20240502 20:46:10.356269 4233 catalog_manager.cc:5484] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2 reported cstate change: config changed from index 4 to 5, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c92e6e46782849479c8a0c6d82276fb2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:10.356761 7299 tablet_replica.cc:330] T c6ac59f330454d2a89ad9c11192ed4f0 P ed47c023452a497192f35d2f617bf9bd: stopping tablet replica
I20240502 20:46:10.356889 7299 raft_consensus.cc:2229] T c6ac59f330454d2a89ad9c11192ed4f0 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:10.357122 7299 raft_consensus.cc:2258] T c6ac59f330454d2a89ad9c11192ed4f0 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:10.357111 4222 catalog_manager.cc:4830] TS a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): tablet 66b8f583dfd24c2aa5a59457a8e58fbb (table rebalance_test_table_18 [id=393c8bbf6b174a9e891d96bcd09fe045]) successfully deleted
I20240502 20:46:10.357582 7299 ts_tablet_manager.cc:1875] T c6ac59f330454d2a89ad9c11192ed4f0 P ed47c023452a497192f35d2f617bf9bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:10.357528 6967 raft_consensus.cc:2932] T c55ba788f592448d8941ff94a8cefc52 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:10.358584 8304 raft_consensus.cc:2932] T 747e7331d7ad4dd4b132672254037e1b P a47e5c56f13843018ac584914a8d4707 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } } }
I20240502 20:46:10.359511 7299 ts_tablet_manager.cc:1888] T c6ac59f330454d2a89ad9c11192ed4f0 P ed47c023452a497192f35d2f617bf9bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 20:46:10.359575 7299 log.cc:1196] T c6ac59f330454d2a89ad9c11192ed4f0 P ed47c023452a497192f35d2f617bf9bd: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal/wals/c6ac59f330454d2a89ad9c11192ed4f0
I20240502 20:46:10.359869 7299 ts_tablet_manager.cc:1909] T c6ac59f330454d2a89ad9c11192ed4f0 P ed47c023452a497192f35d2f617bf9bd: Deleting consensus metadata
I20240502 20:46:10.360498 4222 catalog_manager.cc:4830] TS ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193): tablet c6ac59f330454d2a89ad9c11192ed4f0 (table rebalance_test_table_18 [id=393c8bbf6b174a9e891d96bcd09fe045]) successfully deleted
I20240502 20:46:10.360668 6835 raft_consensus.cc:2932] T 747e7331d7ad4dd4b132672254037e1b P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } } }
I20240502 20:46:10.354784 7297 tablet_replica.cc:330] T a55d5b8b9ed04ca39888b89b82a3f833 P c92e6e46782849479c8a0c6d82276fb2: stopping tablet replica
I20240502 20:46:10.363205 7297 raft_consensus.cc:2229] T a55d5b8b9ed04ca39888b89b82a3f833 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:10.363379 7297 raft_consensus.cc:2258] T a55d5b8b9ed04ca39888b89b82a3f833 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:10.363987 7297 ts_tablet_manager.cc:1875] T a55d5b8b9ed04ca39888b89b82a3f833 P c92e6e46782849479c8a0c6d82276fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:10.365589 4227 catalog_manager.cc:5484] T 747e7331d7ad4dd4b132672254037e1b P a47e5c56f13843018ac584914a8d4707 reported cstate change: config changed from index 2 to 3, c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a47e5c56f13843018ac584914a8d4707" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:10.366245 4374 consensus_queue.cc:237] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } }
I20240502 20:46:10.367619 6967 raft_consensus.cc:1261] T c55ba788f592448d8941ff94a8cefc52 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:10.368048 8918 consensus_queue.cc:1035] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:10.369736 7297 ts_tablet_manager.cc:1888] T a55d5b8b9ed04ca39888b89b82a3f833 P c92e6e46782849479c8a0c6d82276fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240502 20:46:10.369814 7297 log.cc:1196] T a55d5b8b9ed04ca39888b89b82a3f833 P c92e6e46782849479c8a0c6d82276fb2: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal/wals/a55d5b8b9ed04ca39888b89b82a3f833
I20240502 20:46:10.370077 7297 ts_tablet_manager.cc:1909] T a55d5b8b9ed04ca39888b89b82a3f833 P c92e6e46782849479c8a0c6d82276fb2: Deleting consensus metadata
I20240502 20:46:10.370774 4222 catalog_manager.cc:4830] TS c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827): tablet a55d5b8b9ed04ca39888b89b82a3f833 (table rebalance_test_table_18 [id=393c8bbf6b174a9e891d96bcd09fe045]) successfully deleted
I20240502 20:46:10.371409 6957 consensus_queue.cc:237] T 747e7331d7ad4dd4b132672254037e1b P a47e5c56f13843018ac584914a8d4707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } }
I20240502 20:46:10.372658 9159 tablet_bootstrap.cc:492] T 8565ebbd21dd4f708a0254d1ed2d6670 P 7845c397230d49358ba2fad9db53f97c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:10.373054 9159 tablet_bootstrap.cc:492] T 8565ebbd21dd4f708a0254d1ed2d6670 P 7845c397230d49358ba2fad9db53f97c: Bootstrap complete.
I20240502 20:46:10.373086 6835 raft_consensus.cc:1261] T 747e7331d7ad4dd4b132672254037e1b P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Refusing update from remote peer a47e5c56f13843018ac584914a8d4707: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:10.360733 4506 raft_consensus.cc:2932] T 747e7331d7ad4dd4b132672254037e1b P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } } }
I20240502 20:46:10.348973 8141 ts_tablet_manager.cc:1875] T c6ac59f330454d2a89ad9c11192ed4f0 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:10.373529 8528 consensus_queue.cc:1035] T 747e7331d7ad4dd4b132672254037e1b P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:10.374760 4505 raft_consensus.cc:1261] T 747e7331d7ad4dd4b132672254037e1b P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer a47e5c56f13843018ac584914a8d4707: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:10.375430 8141 ts_tablet_manager.cc:1888] T c6ac59f330454d2a89ad9c11192ed4f0 P c3d26eafd51547acb780d4aac5ebe7cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 20:46:10.356652 4507 raft_consensus.cc:2932] T c55ba788f592448d8941ff94a8cefc52 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:10.375517 8141 log.cc:1196] T c6ac59f330454d2a89ad9c11192ed4f0 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal/wals/c6ac59f330454d2a89ad9c11192ed4f0
I20240502 20:46:10.375432 8528 raft_consensus.cc:2932] T 747e7331d7ad4dd4b132672254037e1b P a47e5c56f13843018ac584914a8d4707 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } } }
I20240502 20:46:10.375790 8141 ts_tablet_manager.cc:1909] T c6ac59f330454d2a89ad9c11192ed4f0 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting consensus metadata
I20240502 20:46:10.376171 4221 catalog_manager.cc:4830] TS c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097): tablet c6ac59f330454d2a89ad9c11192ed4f0 (table rebalance_test_table_18 [id=393c8bbf6b174a9e891d96bcd09fe045]) successfully deleted
I20240502 20:46:10.376540 9171 consensus_queue.cc:1035] T 747e7331d7ad4dd4b132672254037e1b P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:10.376708 4508 raft_consensus.cc:1261] T c55ba788f592448d8941ff94a8cefc52 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:10.376876 4222 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 747e7331d7ad4dd4b132672254037e1b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:10.377470 8918 consensus_queue.cc:1035] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:10.377751 4508 raft_consensus.cc:2932] T 747e7331d7ad4dd4b132672254037e1b P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } } }
I20240502 20:46:10.377985 7937 raft_consensus.cc:2932] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:10.379098 6835 raft_consensus.cc:2932] T 747e7331d7ad4dd4b132672254037e1b P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } } }
I20240502 20:46:10.379276 4230 catalog_manager.cc:5484] T 747e7331d7ad4dd4b132672254037e1b P a47e5c56f13843018ac584914a8d4707 reported cstate change: config changed from index 3 to 4, VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) evicted. New cstate: current_term: 1 leader_uuid: "a47e5c56f13843018ac584914a8d4707" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:10.380264 4222 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c55ba788f592448d8941ff94a8cefc52 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:10.380920 6967 raft_consensus.cc:2932] T c55ba788f592448d8941ff94a8cefc52 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:10.380973 4227 catalog_manager.cc:5484] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2 reported cstate change: config changed from index 5 to 6, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New cstate: current_term: 1 leader_uuid: "c92e6e46782849479c8a0c6d82276fb2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:10.377473 9169 tablet_replica.cc:330] T a55d5b8b9ed04ca39888b89b82a3f833 P 7845c397230d49358ba2fad9db53f97c: stopping tablet replica
I20240502 20:46:10.381678 9169 raft_consensus.cc:2229] T a55d5b8b9ed04ca39888b89b82a3f833 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:10.381837 9169 raft_consensus.cc:2258] T a55d5b8b9ed04ca39888b89b82a3f833 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:10.382525 9169 ts_tablet_manager.cc:1875] T a55d5b8b9ed04ca39888b89b82a3f833 P 7845c397230d49358ba2fad9db53f97c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:10.384634 9169 ts_tablet_manager.cc:1888] T a55d5b8b9ed04ca39888b89b82a3f833 P 7845c397230d49358ba2fad9db53f97c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240502 20:46:10.384770 9169 log.cc:1196] T a55d5b8b9ed04ca39888b89b82a3f833 P 7845c397230d49358ba2fad9db53f97c: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/wal/wals/a55d5b8b9ed04ca39888b89b82a3f833
I20240502 20:46:10.385118 9169 ts_tablet_manager.cc:1909] T a55d5b8b9ed04ca39888b89b82a3f833 P 7845c397230d49358ba2fad9db53f97c: Deleting consensus metadata
I20240502 20:46:10.385880 9159 ts_tablet_manager.cc:1367] T 8565ebbd21dd4f708a0254d1ed2d6670 P 7845c397230d49358ba2fad9db53f97c: Time spent bootstrapping tablet: real 0.042s user 0.024s sys 0.004s
I20240502 20:46:10.386404 9159 raft_consensus.cc:348] T 8565ebbd21dd4f708a0254d1ed2d6670 P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } }
I20240502 20:46:10.386652 7017 raft_consensus.cc:1050] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91: attempting to promote NON_VOTER 201d2b6485084e7dad2cbb722af94779 to VOTER
I20240502 20:46:10.386953 9159 raft_consensus.cc:729] T 8565ebbd21dd4f708a0254d1ed2d6670 P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Initialized, Role: LEARNER
I20240502 20:46:10.387132 7017 consensus_queue.cc:237] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } }
I20240502 20:46:10.387152 9159 consensus_queue.cc:260] T 8565ebbd21dd4f708a0254d1ed2d6670 P 7845c397230d49358ba2fad9db53f97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } }
I20240502 20:46:10.388175 9159 ts_tablet_manager.cc:1398] T 8565ebbd21dd4f708a0254d1ed2d6670 P 7845c397230d49358ba2fad9db53f97c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:10.388172 4618 tablet_service.cc:1514] Processing DeleteTablet for tablet c55ba788f592448d8941ff94a8cefc52 with delete_type TABLET_DATA_TOMBSTONED (TS 923cfab926e7443f8ffd7b21ad0a9d91 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57068
I20240502 20:46:10.388422 8510 tablet_replica.cc:330] T c55ba788f592448d8941ff94a8cefc52 P 923cfab926e7443f8ffd7b21ad0a9d91: stopping tablet replica
I20240502 20:46:10.388564 8510 raft_consensus.cc:2229] T c55ba788f592448d8941ff94a8cefc52 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:10.388758 8510 raft_consensus.cc:2258] T c55ba788f592448d8941ff94a8cefc52 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:10.388748 4374 raft_consensus.cc:1261] T eebddad3efe748f78372901ecad5b407 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:10.388958 4658 tablet_copy_service.cc:342] P 923cfab926e7443f8ffd7b21ad0a9d91: Request end of tablet copy session 7845c397230d49358ba2fad9db53f97c-8565ebbd21dd4f708a0254d1ed2d6670 received from {username='slave'} at 127.30.32.69:34201
I20240502 20:46:10.389097 4658 tablet_copy_service.cc:433] P 923cfab926e7443f8ffd7b21ad0a9d91: ending tablet copy session 7845c397230d49358ba2fad9db53f97c-8565ebbd21dd4f708a0254d1ed2d6670 on tablet 8565ebbd21dd4f708a0254d1ed2d6670 with peer 7845c397230d49358ba2fad9db53f97c
I20240502 20:46:10.389329 4768 raft_consensus.cc:1261] T eebddad3efe748f78372901ecad5b407 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:10.389523 8510 ts_tablet_manager.cc:1875] T c55ba788f592448d8941ff94a8cefc52 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:10.389909 7098 raft_consensus.cc:1261] T eebddad3efe748f78372901ecad5b407 P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:10.390012 9042 consensus_queue.cc:1035] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:10.391551 8510 ts_tablet_manager.cc:1888] T c55ba788f592448d8941ff94a8cefc52 P 923cfab926e7443f8ffd7b21ad0a9d91: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:46:10.391628 8510 log.cc:1196] T c55ba788f592448d8941ff94a8cefc52 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal/wals/c55ba788f592448d8941ff94a8cefc52
I20240502 20:46:10.391667 4221 catalog_manager.cc:4830] TS 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): tablet a55d5b8b9ed04ca39888b89b82a3f833 (table rebalance_test_table_18 [id=393c8bbf6b174a9e891d96bcd09fe045]) successfully deleted
I20240502 20:46:10.391774 6681 tablet_service.cc:1514] Processing DeleteTablet for tablet 747e7331d7ad4dd4b132672254037e1b with delete_type TABLET_DATA_TOMBSTONED (TS 7845c397230d49358ba2fad9db53f97c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37506
I20240502 20:46:10.391979 9169 tablet_replica.cc:330] T 747e7331d7ad4dd4b132672254037e1b P 7845c397230d49358ba2fad9db53f97c: stopping tablet replica
I20240502 20:46:10.392270 9169 raft_consensus.cc:2229] T 747e7331d7ad4dd4b132672254037e1b P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:10.392443 9169 pending_rounds.cc:62] T 747e7331d7ad4dd4b132672254037e1b P 7845c397230d49358ba2fad9db53f97c: Trying to abort 1 pending ops.
I20240502 20:46:10.392558 9169 pending_rounds.cc:68] T 747e7331d7ad4dd4b132672254037e1b P 7845c397230d49358ba2fad9db53f97c: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7023340627245735936 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "747e7331d7ad4dd4b132672254037e1b" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } } } }
I20240502 20:46:10.393106 9041 consensus_queue.cc:1035] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:10.393502 9169 raft_consensus.cc:2894] T 747e7331d7ad4dd4b132672254037e1b P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240502 20:46:10.393501 4219 catalog_manager.cc:4830] TS 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141): tablet c55ba788f592448d8941ff94a8cefc52 (table rebalance_test_table_9 [id=7c90eeb05db84afb8f78ab7f14e0ced4]) successfully deleted
I20240502 20:46:10.393627 9169 raft_consensus.cc:2866] T 747e7331d7ad4dd4b132672254037e1b P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 20:46:10.393692 9169 raft_consensus.cc:2258] T 747e7331d7ad4dd4b132672254037e1b P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:10.394176 9041 consensus_queue.cc:1035] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:10.394251 9169 ts_tablet_manager.cc:1875] T 747e7331d7ad4dd4b132672254037e1b P 7845c397230d49358ba2fad9db53f97c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:10.397146 9042 raft_consensus.cc:2932] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:10.398557 9169 ts_tablet_manager.cc:1888] T 747e7331d7ad4dd4b132672254037e1b P 7845c397230d49358ba2fad9db53f97c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 20:46:10.398619 9169 log.cc:1196] T 747e7331d7ad4dd4b132672254037e1b P 7845c397230d49358ba2fad9db53f97c: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/wal/wals/747e7331d7ad4dd4b132672254037e1b
I20240502 20:46:10.399127 4221 catalog_manager.cc:4830] TS 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): tablet 747e7331d7ad4dd4b132672254037e1b (table rebalancer_extra_table_4 [id=1bde7089cdee45b68bdc73a007b74174]) successfully deleted
I20240502 20:46:10.400142 4507 raft_consensus.cc:2932] T c55ba788f592448d8941ff94a8cefc52 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:10.403046 7098 raft_consensus.cc:2932] T eebddad3efe748f78372901ecad5b407 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:10.403694 4230 catalog_manager.cc:5484] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index 4 to 5, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:10.405771 4374 raft_consensus.cc:2932] T eebddad3efe748f78372901ecad5b407 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:10.410459 4768 raft_consensus.cc:2932] T eebddad3efe748f78372901ecad5b407 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:10.412232 4638 consensus_queue.cc:237] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } }
I20240502 20:46:10.413651 7098 raft_consensus.cc:1261] T eebddad3efe748f78372901ecad5b407 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:10.413650 4768 raft_consensus.cc:1261] T eebddad3efe748f78372901ecad5b407 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:10.414060 9041 consensus_queue.cc:1035] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:10.414355 9043 consensus_queue.cc:1035] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:10.415537 9041 raft_consensus.cc:2932] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:10.415941 7098 raft_consensus.cc:2932] T eebddad3efe748f78372901ecad5b407 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:10.417420 8907 raft_consensus.cc:1050] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc: attempting to promote NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 to VOTER
I20240502 20:46:10.417570 4219 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet eebddad3efe748f78372901ecad5b407 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:10.417896 8907 consensus_queue.cc:237] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } }
I20240502 20:46:10.418721 4231 catalog_manager.cc:5484] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index 5 to 6, VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) evicted. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:10.419917 6835 raft_consensus.cc:1261] T 45ade8c672d8445eaffb179772e3759a P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEARNER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:10.420022 4638 raft_consensus.cc:1261] T 45ade8c672d8445eaffb179772e3759a P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:10.420377 7967 consensus_queue.cc:1035] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:10.420280 4768 raft_consensus.cc:2932] T eebddad3efe748f78372901ecad5b407 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:10.420722 7967 consensus_queue.cc:1035] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:10.423734 4768 raft_consensus.cc:1261] T 45ade8c672d8445eaffb179772e3759a P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:10.424297 6375 consensus_queue.cc:1035] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:10.427266 6375 raft_consensus.cc:2932] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:10.428035 4354 tablet_service.cc:1514] Processing DeleteTablet for tablet eebddad3efe748f78372901ecad5b407 with delete_type TABLET_DATA_TOMBSTONED (TS c92e6e46782849479c8a0c6d82276fb2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38306
I20240502 20:46:10.428267 7297 tablet_replica.cc:330] T eebddad3efe748f78372901ecad5b407 P c92e6e46782849479c8a0c6d82276fb2: stopping tablet replica
I20240502 20:46:10.428424 7297 raft_consensus.cc:2229] T eebddad3efe748f78372901ecad5b407 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:10.428597 7297 raft_consensus.cc:2258] T eebddad3efe748f78372901ecad5b407 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:10.429261 4768 raft_consensus.cc:2932] T 45ade8c672d8445eaffb179772e3759a P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:10.429308 6835 raft_consensus.cc:2932] T 45ade8c672d8445eaffb179772e3759a P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:10.430099 7297 ts_tablet_manager.cc:1875] T eebddad3efe748f78372901ecad5b407 P c92e6e46782849479c8a0c6d82276fb2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:10.432556 7297 ts_tablet_manager.cc:1888] T eebddad3efe748f78372901ecad5b407 P c92e6e46782849479c8a0c6d82276fb2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:46:10.432670 7297 log.cc:1196] T eebddad3efe748f78372901ecad5b407 P c92e6e46782849479c8a0c6d82276fb2: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal/wals/eebddad3efe748f78372901ecad5b407
I20240502 20:46:10.432775 4230 catalog_manager.cc:5484] T 45ade8c672d8445eaffb179772e3759a P ed47c023452a497192f35d2f617bf9bd reported cstate change: config changed from index 4 to 5, a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:10.433336 4222 catalog_manager.cc:4830] TS c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827): tablet eebddad3efe748f78372901ecad5b407 (table rebalance_test_table_11 [id=105c28a926194705956796da40cf3dab]) successfully deleted
I20240502 20:46:10.443807 9105 consensus_queue.cc:1035] T ac57ad308b174217ac74c4cb5e1c7285 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:10.444540 4638 raft_consensus.cc:2932] T 45ade8c672d8445eaffb179772e3759a P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:10.446991 4508 consensus_queue.cc:237] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } }
I20240502 20:46:10.448220 4638 raft_consensus.cc:1261] T 45ade8c672d8445eaffb179772e3759a P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:10.448279 6835 raft_consensus.cc:1261] T 45ade8c672d8445eaffb179772e3759a P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:10.448704 8146 consensus_queue.cc:1035] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:10.448904 8907 consensus_queue.cc:1035] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:10.450608 8907 raft_consensus.cc:2932] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:10.451884 4221 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 45ade8c672d8445eaffb179772e3759a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:10.452602 6835 raft_consensus.cc:2932] T 45ade8c672d8445eaffb179772e3759a P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:10.453729 4231 catalog_manager.cc:5484] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: config changed from index 5 to 6, VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) evicted. New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:10.454180 4638 raft_consensus.cc:2932] T 45ade8c672d8445eaffb179772e3759a P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:10.456127 8506 consensus_queue.cc:1035] T ac57ad308b174217ac74c4cb5e1c7285 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:10.456547 4750 tablet_service.cc:1514] Processing DeleteTablet for tablet 45ade8c672d8445eaffb179772e3759a with delete_type TABLET_DATA_TOMBSTONED (TS ed47c023452a497192f35d2f617bf9bd not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38526
I20240502 20:46:10.456785 7299 tablet_replica.cc:330] T 45ade8c672d8445eaffb179772e3759a P ed47c023452a497192f35d2f617bf9bd: stopping tablet replica
I20240502 20:46:10.456921 7299 raft_consensus.cc:2229] T 45ade8c672d8445eaffb179772e3759a P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:10.457104 7299 raft_consensus.cc:2258] T 45ade8c672d8445eaffb179772e3759a P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:10.458776 7299 ts_tablet_manager.cc:1875] T 45ade8c672d8445eaffb179772e3759a P ed47c023452a497192f35d2f617bf9bd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:10.461053 7299 ts_tablet_manager.cc:1888] T 45ade8c672d8445eaffb179772e3759a P ed47c023452a497192f35d2f617bf9bd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:46:10.461133 7299 log.cc:1196] T 45ade8c672d8445eaffb179772e3759a P ed47c023452a497192f35d2f617bf9bd: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal/wals/45ade8c672d8445eaffb179772e3759a
I20240502 20:46:10.461799 4222 catalog_manager.cc:4830] TS ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193): tablet 45ade8c672d8445eaffb179772e3759a (table rebalance_test_table_19 [id=46a52eb51c614964a91ee65db96cf854]) successfully deleted
I20240502 20:46:10.466773 4231 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 20:46:10.466902 4231 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 20:46:10.467264 4638 consensus_queue.cc:237] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } }
I20240502 20:46:10.468914 6703 raft_consensus.cc:1261] T 00eb91e624554baf918e96b098019be8 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:10.468972 4231 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: Sending DeleteTablet for 3 replicas of tablet 1fc0c640a6fb4b24b17642ed26ce8f07
W20240502 20:46:10.469290 4575 consensus_peers.cc:485] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 00eb91e624554baf918e96b098019be8. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:10.469530 6947 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fc0c640a6fb4b24b17642ed26ce8f07 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:10 UTC) from {username='slave'} at 127.0.0.1:45156
I20240502 20:46:10.469650 9043 consensus_queue.cc:1035] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:10.469709 9035 tablet_replica.cc:330] T 1fc0c640a6fb4b24b17642ed26ce8f07 P a47e5c56f13843018ac584914a8d4707: stopping tablet replica
I20240502 20:46:10.469746 4768 raft_consensus.cc:1261] T 00eb91e624554baf918e96b098019be8 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:10.469851 9035 raft_consensus.cc:2229] T 1fc0c640a6fb4b24b17642ed26ce8f07 P a47e5c56f13843018ac584914a8d4707 [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:10.469970 4750 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fc0c640a6fb4b24b17642ed26ce8f07 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:10 UTC) from {username='slave'} at 127.0.0.1:38526
I20240502 20:46:10.470059 9035 raft_consensus.cc:2258] T 1fc0c640a6fb4b24b17642ed26ce8f07 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:10.470309 9043 consensus_queue.cc:1035] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:10.470666 9035 ts_tablet_manager.cc:1875] T 1fc0c640a6fb4b24b17642ed26ce8f07 P a47e5c56f13843018ac584914a8d4707: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:10.470818 7079 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fc0c640a6fb4b24b17642ed26ce8f07 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:10 UTC) from {username='slave'} at 127.0.0.1:59162
I20240502 20:46:10.471082 9191 tablet_replica.cc:330] T 1fc0c640a6fb4b24b17642ed26ce8f07 P 201d2b6485084e7dad2cbb722af94779: stopping tablet replica
I20240502 20:46:10.471215 9191 raft_consensus.cc:2229] T 1fc0c640a6fb4b24b17642ed26ce8f07 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:10.471436 9041 raft_consensus.cc:2932] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } } }
I20240502 20:46:10.472277 9191 raft_consensus.cc:2258] T 1fc0c640a6fb4b24b17642ed26ce8f07 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:10.472587 9035 ts_tablet_manager.cc:1888] T 1fc0c640a6fb4b24b17642ed26ce8f07 P a47e5c56f13843018ac584914a8d4707: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:10.472647 9035 log.cc:1196] T 1fc0c640a6fb4b24b17642ed26ce8f07 P a47e5c56f13843018ac584914a8d4707: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/wal/wals/1fc0c640a6fb4b24b17642ed26ce8f07
I20240502 20:46:10.472908 9035 ts_tablet_manager.cc:1909] T 1fc0c640a6fb4b24b17642ed26ce8f07 P a47e5c56f13843018ac584914a8d4707: Deleting consensus metadata
I20240502 20:46:10.473310 9191 ts_tablet_manager.cc:1875] T 1fc0c640a6fb4b24b17642ed26ce8f07 P 201d2b6485084e7dad2cbb722af94779: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:10.473474 6703 raft_consensus.cc:2932] T 00eb91e624554baf918e96b098019be8 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } } }
I20240502 20:46:10.473937 7299 tablet_replica.cc:330] T 1fc0c640a6fb4b24b17642ed26ce8f07 P ed47c023452a497192f35d2f617bf9bd: stopping tablet replica
I20240502 20:46:10.474115 7299 raft_consensus.cc:2229] T 1fc0c640a6fb4b24b17642ed26ce8f07 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:10.474277 7299 raft_consensus.cc:2258] T 1fc0c640a6fb4b24b17642ed26ce8f07 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:10.474264 4231 catalog_manager.cc:5484] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index 6 to 7, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:10.474856 7299 ts_tablet_manager.cc:1875] T 1fc0c640a6fb4b24b17642ed26ce8f07 P ed47c023452a497192f35d2f617bf9bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:10.475632 4222 catalog_manager.cc:4830] TS a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): tablet 1fc0c640a6fb4b24b17642ed26ce8f07 (table rebalancer_dynamic_table_new_name [id=44b9d8eb907c483bb3ba9399264a26fb]) successfully deleted
I20240502 20:46:10.476769 7299 ts_tablet_manager.cc:1888] T 1fc0c640a6fb4b24b17642ed26ce8f07 P ed47c023452a497192f35d2f617bf9bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:10.476838 7299 log.cc:1196] T 1fc0c640a6fb4b24b17642ed26ce8f07 P ed47c023452a497192f35d2f617bf9bd: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal/wals/1fc0c640a6fb4b24b17642ed26ce8f07
I20240502 20:46:10.477113 7299 ts_tablet_manager.cc:1909] T 1fc0c640a6fb4b24b17642ed26ce8f07 P ed47c023452a497192f35d2f617bf9bd: Deleting consensus metadata
I20240502 20:46:10.477696 4222 catalog_manager.cc:4830] TS ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193): tablet 1fc0c640a6fb4b24b17642ed26ce8f07 (table rebalancer_dynamic_table_new_name [id=44b9d8eb907c483bb3ba9399264a26fb]) successfully deleted
I20240502 20:46:10.480701 4768 raft_consensus.cc:2932] T 00eb91e624554baf918e96b098019be8 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } } }
I20240502 20:46:10.481274 9191 ts_tablet_manager.cc:1888] T 1fc0c640a6fb4b24b17642ed26ce8f07 P 201d2b6485084e7dad2cbb722af94779: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:10.481351 9191 log.cc:1196] T 1fc0c640a6fb4b24b17642ed26ce8f07 P 201d2b6485084e7dad2cbb722af94779: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/wal/wals/1fc0c640a6fb4b24b17642ed26ce8f07
I20240502 20:46:10.481572 9191 ts_tablet_manager.cc:1909] T 1fc0c640a6fb4b24b17642ed26ce8f07 P 201d2b6485084e7dad2cbb722af94779: Deleting consensus metadata
I20240502 20:46:10.482381 4219 catalog_manager.cc:4830] TS 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): tablet 1fc0c640a6fb4b24b17642ed26ce8f07 (table rebalancer_dynamic_table_new_name [id=44b9d8eb907c483bb3ba9399264a26fb]) successfully deleted
I20240502 20:46:10.488613 4507 consensus_queue.cc:237] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } }
I20240502 20:46:10.490132 4638 raft_consensus.cc:1261] T 41f84d5db2034af591b56e109be67934 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:10.490324 4374 raft_consensus.cc:1261] T 41f84d5db2034af591b56e109be67934 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:10.490531 8146 consensus_queue.cc:1035] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 20:46:10.490726 4441 consensus_peers.cc:485] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41f84d5db2034af591b56e109be67934. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:10.490774 6375 consensus_queue.cc:1035] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:10.492689 7967 raft_consensus.cc:2932] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } } }
I20240502 20:46:10.493166 4638 raft_consensus.cc:2932] T 41f84d5db2034af591b56e109be67934 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } } }
I20240502 20:46:10.495494 4374 raft_consensus.cc:2932] T 41f84d5db2034af591b56e109be67934 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } } }
I20240502 20:46:10.495915 4227 catalog_manager.cc:5484] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: config changed from index -1 to 4, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:10.503265 4638 consensus_queue.cc:237] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } }
I20240502 20:46:10.504583 4374 raft_consensus.cc:1261] T 02302fc79dc04b0396041c600c2f4bef P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:10.504635 4507 raft_consensus.cc:1261] T 02302fc79dc04b0396041c600c2f4bef P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:10.505004 9043 consensus_queue.cc:1035] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 20:46:10.505182 4574 consensus_peers.cc:485] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 02302fc79dc04b0396041c600c2f4bef. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:10.505283 8236 consensus_queue.cc:1035] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:10.506842 7017 raft_consensus.cc:2932] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } } }
I20240502 20:46:10.507301 4374 raft_consensus.cc:2932] T 02302fc79dc04b0396041c600c2f4bef P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } } }
I20240502 20:46:10.509456 4230 catalog_manager.cc:5484] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index -1 to 4, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:10.510000 4507 raft_consensus.cc:2932] T 02302fc79dc04b0396041c600c2f4bef P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } } }
I20240502 20:46:10.519420 4638 consensus_queue.cc:237] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } }
I20240502 20:46:10.520658 4768 raft_consensus.cc:1261] T 28da7347f982468281570ffa7503a854 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:10.520691 4507 raft_consensus.cc:1261] T 28da7347f982468281570ffa7503a854 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:10.521065 7017 consensus_queue.cc:1035] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:10.521278 9041 consensus_queue.cc:1035] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 20:46:10.521481 4573 consensus_peers.cc:485] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867): Couldn't send request to peer a4bf1df7fa6b4948be502c182bfbb833. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 28da7347f982468281570ffa7503a854. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:10.523056 9043 raft_consensus.cc:2932] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } } }
I20240502 20:46:10.523499 4507 raft_consensus.cc:2932] T 28da7347f982468281570ffa7503a854 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } } }
I20240502 20:46:10.523537 4768 raft_consensus.cc:2932] T 28da7347f982468281570ffa7503a854 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } } }
I20240502 20:46:10.525594 4227 catalog_manager.cc:5484] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index -1 to 4, NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) added. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:10.571112 4230 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41812:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 20:46:10.576593 4354 tablet_service.cc:1467] Processing CreateTablet for tablet 8eebae62c8054dc48e485b7e45daeffa (DEFAULT_TABLE table=rebalancer_dynamic_table [id=83a915ef71d94c1da0ade39f98e82b29]), partition=
I20240502 20:46:10.576892 4354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eebae62c8054dc48e485b7e45daeffa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:10.577024 6681 tablet_service.cc:1467] Processing CreateTablet for tablet 8eebae62c8054dc48e485b7e45daeffa (DEFAULT_TABLE table=rebalancer_dynamic_table [id=83a915ef71d94c1da0ade39f98e82b29]), partition=
I20240502 20:46:10.577267 6681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eebae62c8054dc48e485b7e45daeffa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:10.579380 9262 tablet_bootstrap.cc:492] T 8eebae62c8054dc48e485b7e45daeffa P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:46:10.580111 4618 tablet_service.cc:1467] Processing CreateTablet for tablet 8eebae62c8054dc48e485b7e45daeffa (DEFAULT_TABLE table=rebalancer_dynamic_table [id=83a915ef71d94c1da0ade39f98e82b29]), partition=
I20240502 20:46:10.580349 4618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eebae62c8054dc48e485b7e45daeffa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:10.580403 9262 tablet_bootstrap.cc:654] T 8eebae62c8054dc48e485b7e45daeffa P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:10.581487 9262 tablet_bootstrap.cc:492] T 8eebae62c8054dc48e485b7e45daeffa P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:46:10.581663 9262 ts_tablet_manager.cc:1367] T 8eebae62c8054dc48e485b7e45daeffa P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:10.581771 8506 consensus_queue.cc:1035] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:10.582056 9262 raft_consensus.cc:348] T 8eebae62c8054dc48e485b7e45daeffa P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:10.582176 9262 raft_consensus.cc:374] T 8eebae62c8054dc48e485b7e45daeffa P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:10.582209 9262 raft_consensus.cc:729] T 8eebae62c8054dc48e485b7e45daeffa P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:46:10.582368 9262 consensus_queue.cc:260] T 8eebae62c8054dc48e485b7e45daeffa P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:10.582666 9262 ts_tablet_manager.cc:1398] T 8eebae62c8054dc48e485b7e45daeffa P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:10.586519 9041 raft_consensus.cc:1050] T dbc1ef85e30d438682bc86935ceefd1b P 923cfab926e7443f8ffd7b21ad0a9d91: attempting to promote NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 to VOTER
I20240502 20:46:10.587028 9041 consensus_queue.cc:237] T dbc1ef85e30d438682bc86935ceefd1b P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } }
I20240502 20:46:10.588943 4374 raft_consensus.cc:1261] T dbc1ef85e30d438682bc86935ceefd1b P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:10.589015 4507 raft_consensus.cc:1261] T dbc1ef85e30d438682bc86935ceefd1b P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:10.589427 6835 raft_consensus.cc:1261] T dbc1ef85e30d438682bc86935ceefd1b P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEARNER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:10.589500 8168 consensus_queue.cc:1035] T dbc1ef85e30d438682bc86935ceefd1b P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:10.589788 7017 consensus_queue.cc:1035] T dbc1ef85e30d438682bc86935ceefd1b P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:10.590045 8168 consensus_queue.cc:1035] T dbc1ef85e30d438682bc86935ceefd1b P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:10.591673 9105 consensus_queue.cc:1035] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:10.594218 9041 raft_consensus.cc:2932] T dbc1ef85e30d438682bc86935ceefd1b P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:10.594910 6834 raft_consensus.cc:2932] T dbc1ef85e30d438682bc86935ceefd1b P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:10.596397 4507 raft_consensus.cc:2932] T dbc1ef85e30d438682bc86935ceefd1b P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:10.597079 4228 catalog_manager.cc:5484] T dbc1ef85e30d438682bc86935ceefd1b P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index 4 to 5, a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:10.600265 4638 consensus_queue.cc:237] T dbc1ef85e30d438682bc86935ceefd1b P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } }
I20240502 20:46:10.601362 6834 raft_consensus.cc:1261] T dbc1ef85e30d438682bc86935ceefd1b P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:10.601701 7017 consensus_queue.cc:1035] T dbc1ef85e30d438682bc86935ceefd1b P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:10.601804 4507 raft_consensus.cc:1261] T dbc1ef85e30d438682bc86935ceefd1b P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:10.602325 7017 consensus_queue.cc:1035] T dbc1ef85e30d438682bc86935ceefd1b P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:10.602782 9263 tablet_bootstrap.cc:492] T 8eebae62c8054dc48e485b7e45daeffa P 7845c397230d49358ba2fad9db53f97c: Bootstrap starting.
I20240502 20:46:10.605438 7017 raft_consensus.cc:2932] T dbc1ef85e30d438682bc86935ceefd1b P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:10.607249 4219 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dbc1ef85e30d438682bc86935ceefd1b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:10.607096 4507 raft_consensus.cc:2932] T dbc1ef85e30d438682bc86935ceefd1b P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:10.607432 6834 raft_consensus.cc:2932] T dbc1ef85e30d438682bc86935ceefd1b P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:10.608196 9263 tablet_bootstrap.cc:654] T 8eebae62c8054dc48e485b7e45daeffa P 7845c397230d49358ba2fad9db53f97c: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:10.608960 4231 catalog_manager.cc:5484] T dbc1ef85e30d438682bc86935ceefd1b P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index 5 to 6, VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) evicted. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:10.609797 9265 tablet_bootstrap.cc:492] T 8eebae62c8054dc48e485b7e45daeffa P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:46:10.612658 9265 tablet_bootstrap.cc:654] T 8eebae62c8054dc48e485b7e45daeffa P 923cfab926e7443f8ffd7b21ad0a9d91: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:10.612953 4354 tablet_service.cc:1514] Processing DeleteTablet for tablet dbc1ef85e30d438682bc86935ceefd1b with delete_type TABLET_DATA_TOMBSTONED (TS c92e6e46782849479c8a0c6d82276fb2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38306
I20240502 20:46:10.613282 7297 tablet_replica.cc:330] T dbc1ef85e30d438682bc86935ceefd1b P c92e6e46782849479c8a0c6d82276fb2: stopping tablet replica
I20240502 20:46:10.613449 7297 raft_consensus.cc:2229] T dbc1ef85e30d438682bc86935ceefd1b P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:10.613610 7297 pending_rounds.cc:62] T dbc1ef85e30d438682bc86935ceefd1b P c92e6e46782849479c8a0c6d82276fb2: Trying to abort 1 pending ops.
I20240502 20:46:10.613682 7297 pending_rounds.cc:68] T dbc1ef85e30d438682bc86935ceefd1b P c92e6e46782849479c8a0c6d82276fb2: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023340628323082240 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "dbc1ef85e30d438682bc86935ceefd1b" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } } }
I20240502 20:46:10.613883 7297 raft_consensus.cc:2894] T dbc1ef85e30d438682bc86935ceefd1b P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240502 20:46:10.613955 7297 raft_consensus.cc:2866] T dbc1ef85e30d438682bc86935ceefd1b P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 20:46:10.614002 7297 raft_consensus.cc:2258] T dbc1ef85e30d438682bc86935ceefd1b P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 20:46:10.615228 9269 consensus_queue.cc:1159] T dbc1ef85e30d438682bc86935ceefd1b P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "c92e6e46782849479c8a0c6d82276fb2" status { last_received { term: 1 index: 5 } last_committed_idx: 4 last_received_current_leader { term: 1 index: 5 } }
I20240502 20:46:10.615696 9265 tablet_bootstrap.cc:492] T 8eebae62c8054dc48e485b7e45daeffa P 923cfab926e7443f8ffd7b21ad0a9d91: No bootstrap required, opened a new log
I20240502 20:46:10.615787 9265 ts_tablet_manager.cc:1367] T 8eebae62c8054dc48e485b7e45daeffa P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 20:46:10.616220 9265 raft_consensus.cc:348] T 8eebae62c8054dc48e485b7e45daeffa P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:10.616454 9265 raft_consensus.cc:374] T 8eebae62c8054dc48e485b7e45daeffa P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:10.616545 9265 raft_consensus.cc:729] T 8eebae62c8054dc48e485b7e45daeffa P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: FOLLOWER
I20240502 20:46:10.616714 9265 consensus_queue.cc:260] T 8eebae62c8054dc48e485b7e45daeffa P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:10.617527 9265 ts_tablet_manager.cc:1398] T 8eebae62c8054dc48e485b7e45daeffa P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:10.618295 7297 ts_tablet_manager.cc:1875] T dbc1ef85e30d438682bc86935ceefd1b P c92e6e46782849479c8a0c6d82276fb2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:10.619074 9263 tablet_bootstrap.cc:492] T 8eebae62c8054dc48e485b7e45daeffa P 7845c397230d49358ba2fad9db53f97c: No bootstrap required, opened a new log
I20240502 20:46:10.619164 9263 ts_tablet_manager.cc:1367] T 8eebae62c8054dc48e485b7e45daeffa P 7845c397230d49358ba2fad9db53f97c: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240502 20:46:10.619743 9263 raft_consensus.cc:348] T 8eebae62c8054dc48e485b7e45daeffa P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:10.619870 9263 raft_consensus.cc:374] T 8eebae62c8054dc48e485b7e45daeffa P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:10.619911 9263 raft_consensus.cc:729] T 8eebae62c8054dc48e485b7e45daeffa P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Initialized, Role: FOLLOWER
I20240502 20:46:10.620115 9263 consensus_queue.cc:260] T 8eebae62c8054dc48e485b7e45daeffa P 7845c397230d49358ba2fad9db53f97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:10.620450 9263 ts_tablet_manager.cc:1398] T 8eebae62c8054dc48e485b7e45daeffa P 7845c397230d49358ba2fad9db53f97c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:10.620764 7297 ts_tablet_manager.cc:1888] T dbc1ef85e30d438682bc86935ceefd1b P c92e6e46782849479c8a0c6d82276fb2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:46:10.620817 7297 log.cc:1196] T dbc1ef85e30d438682bc86935ceefd1b P c92e6e46782849479c8a0c6d82276fb2: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal/wals/dbc1ef85e30d438682bc86935ceefd1b
I20240502 20:46:10.621093 7937 raft_consensus.cc:482] T 8eebae62c8054dc48e485b7e45daeffa P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:10.621203 7937 raft_consensus.cc:504] T 8eebae62c8054dc48e485b7e45daeffa P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:10.621583 7937 leader_election.cc:288] T 8eebae62c8054dc48e485b7e45daeffa P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:46:10.621991 6703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eebae62c8054dc48e485b7e45daeffa" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7845c397230d49358ba2fad9db53f97c" is_pre_election: true
I20240502 20:46:10.622109 6703 raft_consensus.cc:2454] T 8eebae62c8054dc48e485b7e45daeffa P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 0.
I20240502 20:46:10.622439 4312 leader_election.cc:302] T 8eebae62c8054dc48e485b7e45daeffa P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7845c397230d49358ba2fad9db53f97c, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:46:10.622637 7937 raft_consensus.cc:2781] T 8eebae62c8054dc48e485b7e45daeffa P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:10.622689 7937 raft_consensus.cc:482] T 8eebae62c8054dc48e485b7e45daeffa P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:10.622731 7937 raft_consensus.cc:3037] T 8eebae62c8054dc48e485b7e45daeffa P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:10.622759 4638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eebae62c8054dc48e485b7e45daeffa" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" is_pre_election: true
I20240502 20:46:10.622884 4638 raft_consensus.cc:2454] T 8eebae62c8054dc48e485b7e45daeffa P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 0.
I20240502 20:46:10.623708 7937 raft_consensus.cc:504] T 8eebae62c8054dc48e485b7e45daeffa P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:10.623896 4222 catalog_manager.cc:4830] TS c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827): tablet dbc1ef85e30d438682bc86935ceefd1b (table rebalance_test_table_3 [id=54d5208bcf1c43998e3ac81026ea9fbe]) successfully deleted
I20240502 20:46:10.624099 7937 leader_election.cc:288] T 8eebae62c8054dc48e485b7e45daeffa P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 election: Requested vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:46:10.624526 4638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eebae62c8054dc48e485b7e45daeffa" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91"
I20240502 20:46:10.624627 4638 raft_consensus.cc:3037] T 8eebae62c8054dc48e485b7e45daeffa P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:10.624866 6703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eebae62c8054dc48e485b7e45daeffa" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7845c397230d49358ba2fad9db53f97c"
I20240502 20:46:10.625000 6703 raft_consensus.cc:3037] T 8eebae62c8054dc48e485b7e45daeffa P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:10.625967 6703 raft_consensus.cc:2454] T 8eebae62c8054dc48e485b7e45daeffa P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 1.
I20240502 20:46:10.625970 4638 raft_consensus.cc:2454] T 8eebae62c8054dc48e485b7e45daeffa P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 1.
I20240502 20:46:10.626266 4312 leader_election.cc:302] T 8eebae62c8054dc48e485b7e45daeffa P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7845c397230d49358ba2fad9db53f97c, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:46:10.626459 7937 raft_consensus.cc:2781] T 8eebae62c8054dc48e485b7e45daeffa P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:10.626534 7937 raft_consensus.cc:686] T 8eebae62c8054dc48e485b7e45daeffa P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEADER]: Becoming Leader. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Running, Role: LEADER
I20240502 20:46:10.626657 7937 consensus_queue.cc:237] T 8eebae62c8054dc48e485b7e45daeffa P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:10.629011 4230 catalog_manager.cc:5484] T 8eebae62c8054dc48e485b7e45daeffa P c92e6e46782849479c8a0c6d82276fb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65). New cstate: current_term: 1 leader_uuid: "c92e6e46782849479c8a0c6d82276fb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:10.637658 9105 consensus_queue.cc:1035] T 77dc51e6a85a4ddcb64db12b40c5810f P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:10.642135 9297 consensus_queue.cc:1035] T 77dc51e6a85a4ddcb64db12b40c5810f P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:10.646463 7099 raft_consensus.cc:1203] T d6fb394fdcff436886fe0634d79c024c P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 20:46:10.680778 4507 consensus_queue.cc:237] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } }
I20240502 20:46:10.682411 6703 raft_consensus.cc:1261] T a62a7ce526f14c25994bb9e0e3c3bc84 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 20:46:10.682667 4442 consensus_peers.cc:485] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867): Couldn't send request to peer a4bf1df7fa6b4948be502c182bfbb833. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a62a7ce526f14c25994bb9e0e3c3bc84. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:10.682781 6375 consensus_queue.cc:1035] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:10.683802 4374 raft_consensus.cc:1261] T a62a7ce526f14c25994bb9e0e3c3bc84 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:10.684217 6375 consensus_queue.cc:1035] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:10.684266 4638 raft_consensus.cc:1203] T 15a2cbe8285a4938be7f628f7f8b734d P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 20:46:10.684821 6375 raft_consensus.cc:2932] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } } }
I20240502 20:46:10.686853 6703 raft_consensus.cc:2932] T a62a7ce526f14c25994bb9e0e3c3bc84 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } } }
I20240502 20:46:10.687508 4230 catalog_manager.cc:5484] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: config changed from index 6 to 7, NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) added. New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:10.689846 4374 raft_consensus.cc:2932] T a62a7ce526f14c25994bb9e0e3c3bc84 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } } }
I20240502 20:46:10.692804 9042 raft_consensus.cc:1050] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91: attempting to promote NON_VOTER a47e5c56f13843018ac584914a8d4707 to VOTER
I20240502 20:46:10.693332 9042 consensus_queue.cc:237] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } }
I20240502 20:46:10.695039 6834 raft_consensus.cc:1261] T 7abfd61f55554042978b50761cbd6ab5 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:10.695098 4768 raft_consensus.cc:1261] T 7abfd61f55554042978b50761cbd6ab5 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:10.695207 6967 raft_consensus.cc:1261] T 7abfd61f55554042978b50761cbd6ab5 P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:10.695633 9043 consensus_queue.cc:1035] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:10.695894 9041 consensus_queue.cc:1035] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:10.696427 9043 consensus_queue.cc:1035] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:10.700023 8236 raft_consensus.cc:2932] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:10.701659 4768 consensus_queue.cc:237] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } }
I20240502 20:46:10.702073 6834 raft_consensus.cc:2932] T 7abfd61f55554042978b50761cbd6ab5 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:10.702912 4231 catalog_manager.cc:5484] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index 7 to 8, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:10.703193 4374 raft_consensus.cc:1261] T 20373b53281a4214b27f014396f7a651 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:10.703763 4638 raft_consensus.cc:1261] T 20373b53281a4214b27f014396f7a651 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 20:46:10.704560 4703 consensus_peers.cc:485] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 20373b53281a4214b27f014396f7a651. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:10.704528 6967 raft_consensus.cc:2932] T 7abfd61f55554042978b50761cbd6ab5 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:10.704741 9069 consensus_queue.cc:1035] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:10.705046 4768 raft_consensus.cc:2932] T 7abfd61f55554042978b50761cbd6ab5 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:10.705467 9125 consensus_queue.cc:1035] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:10.706861 4638 consensus_queue.cc:237] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } }
I20240502 20:46:10.708122 6834 raft_consensus.cc:1261] T 7abfd61f55554042978b50761cbd6ab5 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:46:10.708529 8236 consensus_queue.cc:1035] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:10.708611 6967 raft_consensus.cc:1261] T 7abfd61f55554042978b50761cbd6ab5 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:46:10.710330 8236 consensus_queue.cc:1035] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:10.710633 9042 raft_consensus.cc:2932] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:10.711033 6835 raft_consensus.cc:2932] T 7abfd61f55554042978b50761cbd6ab5 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:10.712524 6967 raft_consensus.cc:2932] T 7abfd61f55554042978b50761cbd6ab5 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:10.713230 4219 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7abfd61f55554042978b50761cbd6ab5 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:10.714540 4231 catalog_manager.cc:5484] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index 8 to 9, VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) evicted. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:10.716531 4637 raft_consensus.cc:2932] T 20373b53281a4214b27f014396f7a651 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } } }
I20240502 20:46:10.715996 9343 raft_consensus.cc:2932] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } } }
I20240502 20:46:10.716579 4374 raft_consensus.cc:2932] T 20373b53281a4214b27f014396f7a651 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } } }
I20240502 20:46:10.719020 4750 tablet_service.cc:1514] Processing DeleteTablet for tablet 7abfd61f55554042978b50761cbd6ab5 with delete_type TABLET_DATA_TOMBSTONED (TS ed47c023452a497192f35d2f617bf9bd not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:38526
I20240502 20:46:10.719256 7299 tablet_replica.cc:330] T 7abfd61f55554042978b50761cbd6ab5 P ed47c023452a497192f35d2f617bf9bd: stopping tablet replica
I20240502 20:46:10.719399 7299 raft_consensus.cc:2229] T 7abfd61f55554042978b50761cbd6ab5 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:10.719625 7299 raft_consensus.cc:2258] T 7abfd61f55554042978b50761cbd6ab5 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:10.719940 4230 catalog_manager.cc:5484] T 20373b53281a4214b27f014396f7a651 P c92e6e46782849479c8a0c6d82276fb2 reported cstate change: config changed from index -1 to 4, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } } }
I20240502 20:46:10.721107 7299 ts_tablet_manager.cc:1875] T 7abfd61f55554042978b50761cbd6ab5 P ed47c023452a497192f35d2f617bf9bd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:10.723157 7299 ts_tablet_manager.cc:1888] T 7abfd61f55554042978b50761cbd6ab5 P ed47c023452a497192f35d2f617bf9bd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 20:46:10.723234 7299 log.cc:1196] T 7abfd61f55554042978b50761cbd6ab5 P ed47c023452a497192f35d2f617bf9bd: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal/wals/7abfd61f55554042978b50761cbd6ab5
I20240502 20:46:10.723745 4222 catalog_manager.cc:4830] TS ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193): tablet 7abfd61f55554042978b50761cbd6ab5 (table rebalance_test_table_13 [id=c22b8adbea7b4dc49a4794a5d8f81f9b]) successfully deleted
I20240502 20:46:10.727330 6703 raft_consensus.cc:1203] T 8565ebbd21dd4f708a0254d1ed2d6670 P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 20:46:10.732051 4637 consensus_queue.cc:237] T d6c2626b3ee7493c8a349f08e17082b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } }
W20240502 20:46:10.733654 4574 consensus_peers.cc:485] T d6c2626b3ee7493c8a349f08e17082b5 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097): Couldn't send request to peer c3d26eafd51547acb780d4aac5ebe7cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d6c2626b3ee7493c8a349f08e17082b5. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:10.733793 6834 raft_consensus.cc:1261] T d6c2626b3ee7493c8a349f08e17082b5 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:10.733906 6967 raft_consensus.cc:1261] T d6c2626b3ee7493c8a349f08e17082b5 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:10.734092 4227 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 20:46:10.734313 9043 consensus_queue.cc:1035] T d6c2626b3ee7493c8a349f08e17082b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:46:10.735037 9043 consensus_queue.cc:1035] T d6c2626b3ee7493c8a349f08e17082b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:46:10.736160 8236 raft_consensus.cc:2932] T d6c2626b3ee7493c8a349f08e17082b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } } }
I20240502 20:46:10.738124 6967 raft_consensus.cc:2932] T d6c2626b3ee7493c8a349f08e17082b5 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } } }
I20240502 20:46:10.738926 6834 raft_consensus.cc:2932] T d6c2626b3ee7493c8a349f08e17082b5 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } } }
I20240502 20:46:10.739122 4227 catalog_manager.cc:5484] T d6c2626b3ee7493c8a349f08e17082b5 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index -1 to 2, NON_VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) added. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:10.740918 6703 raft_consensus.cc:1261] T 8eebae62c8054dc48e485b7e45daeffa P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:10.741585 4637 raft_consensus.cc:1261] T 8eebae62c8054dc48e485b7e45daeffa P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:10.741689 6138 consensus_queue.cc:1035] T 8eebae62c8054dc48e485b7e45daeffa P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:10.743721 7937 consensus_queue.cc:1035] T 8eebae62c8054dc48e485b7e45daeffa P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:10.744776 4228 catalog_manager.cc:5576] TS 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429) does not have the latest schema for tablet 8eebae62c8054dc48e485b7e45daeffa (table rebalancer_dynamic_table [id=83a915ef71d94c1da0ade39f98e82b29]). Expected version 1 got 0
I20240502 20:46:10.746616 4228 catalog_manager.cc:5576] TS c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827) does not have the latest schema for tablet 8eebae62c8054dc48e485b7e45daeffa (table rebalancer_dynamic_table [id=83a915ef71d94c1da0ade39f98e82b29]). Expected version 1 got 0
I20240502 20:46:10.747855 9364 tablet.cc:1722] T 8eebae62c8054dc48e485b7e45daeffa P 7845c397230d49358ba2fad9db53f97c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 20:46:10.748261 9364 tablet.cc:1620] T 8eebae62c8054dc48e485b7e45daeffa P 7845c397230d49358ba2fad9db53f97c: MemRowSet was empty: no flush needed.
I20240502 20:46:10.748813 4233 catalog_manager.cc:5576] TS c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827) does not have the latest schema for tablet 8eebae62c8054dc48e485b7e45daeffa (table rebalancer_dynamic_table [id=83a915ef71d94c1da0ade39f98e82b29]). Expected version 1 got 0
I20240502 20:46:10.750464 4228 catalog_manager.cc:5576] TS 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429) does not have the latest schema for tablet 8eebae62c8054dc48e485b7e45daeffa (table rebalancer_dynamic_table [id=83a915ef71d94c1da0ade39f98e82b29]). Expected version 1 got 0
I20240502 20:46:10.751492 4231 catalog_manager.cc:5576] TS 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141) does not have the latest schema for tablet 8eebae62c8054dc48e485b7e45daeffa (table rebalancer_dynamic_table [id=83a915ef71d94c1da0ade39f98e82b29]). Expected version 1 got 0
I20240502 20:46:10.753629 9357 tablet.cc:1722] T 8eebae62c8054dc48e485b7e45daeffa P c92e6e46782849479c8a0c6d82276fb2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 20:46:10.753820 9363 tablet.cc:1722] T 8eebae62c8054dc48e485b7e45daeffa P 923cfab926e7443f8ffd7b21ad0a9d91: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 20:46:10.754177 9363 tablet.cc:1620] T 8eebae62c8054dc48e485b7e45daeffa P 923cfab926e7443f8ffd7b21ad0a9d91: MemRowSet was empty: no flush needed.
I20240502 20:46:10.754635 9357 tablet.cc:1620] T 8eebae62c8054dc48e485b7e45daeffa P c92e6e46782849479c8a0c6d82276fb2: MemRowSet was empty: no flush needed.
I20240502 20:46:10.755724 4231 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: rebalancer_dynamic_table [id=83a915ef71d94c1da0ade39f98e82b29] alter complete (version 1)
I20240502 20:46:10.760131 6967 consensus_queue.cc:237] T ef4002126e8847809add182f262d62d9 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: true } }
I20240502 20:46:10.761564 6834 raft_consensus.cc:1261] T ef4002126e8847809add182f262d62d9 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Refusing update from remote peer a47e5c56f13843018ac584914a8d4707: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:10.761651 4374 raft_consensus.cc:1261] T ef4002126e8847809add182f262d62d9 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer a47e5c56f13843018ac584914a8d4707: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:10.761957 8965 consensus_queue.cc:1035] T ef4002126e8847809add182f262d62d9 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 20:46:10.762162 6903 consensus_peers.cc:485] T ef4002126e8847809add182f262d62d9 P a47e5c56f13843018ac584914a8d4707 -> Peer ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193): Couldn't send request to peer ed47c023452a497192f35d2f617bf9bd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ef4002126e8847809add182f262d62d9. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:10.762311 8965 consensus_queue.cc:1035] T ef4002126e8847809add182f262d62d9 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:46:10.763804 8965 raft_consensus.cc:2932] T ef4002126e8847809add182f262d62d9 P a47e5c56f13843018ac584914a8d4707 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: true } } }
I20240502 20:46:10.764251 6834 raft_consensus.cc:2932] T ef4002126e8847809add182f262d62d9 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: true } } }
I20240502 20:46:10.764513 4374 raft_consensus.cc:2932] T ef4002126e8847809add182f262d62d9 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: true } } }
I20240502 20:46:10.767045 4231 catalog_manager.cc:5484] T ef4002126e8847809add182f262d62d9 P a47e5c56f13843018ac584914a8d4707 reported cstate change: config changed from index -1 to 2, NON_VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) added. New cstate: current_term: 1 leader_uuid: "a47e5c56f13843018ac584914a8d4707" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:10.775365 4374 consensus_queue.cc:237] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } }
I20240502 20:46:10.776616 6967 raft_consensus.cc:1261] T 93b7f7de1f8c49389c3d8c1e03c67f26 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:10.776679 6834 raft_consensus.cc:1261] T 93b7f7de1f8c49389c3d8c1e03c67f26 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 20:46:10.776901 4312 consensus_peers.cc:485] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 93b7f7de1f8c49389c3d8c1e03c67f26. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:10.777118 6138 consensus_queue.cc:1035] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:46:10.777288 8918 consensus_queue.cc:1035] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:46:10.778806 8185 raft_consensus.cc:2932] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } } }
I20240502 20:46:10.779280 6834 raft_consensus.cc:2932] T 93b7f7de1f8c49389c3d8c1e03c67f26 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } } }
I20240502 20:46:10.779381 6967 raft_consensus.cc:2932] T 93b7f7de1f8c49389c3d8c1e03c67f26 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } } }
I20240502 20:46:10.782987 4228 catalog_manager.cc:5484] T 93b7f7de1f8c49389c3d8c1e03c67f26 P a4bf1df7fa6b4948be502c182bfbb833 reported cstate change: config changed from index -1 to 2, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New cstate: current_term: 1 leader_uuid: "c92e6e46782849479c8a0c6d82276fb2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } } }
I20240502 20:46:10.784657 4230 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41812:
name: "rebalancer_extra_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:46:10.790657 7079 tablet_service.cc:1467] Processing CreateTablet for tablet 51232fccd98c461fb3d9016081f52ba5 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=90e5baaad90e44bc8ebd55fcb5256a71]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:10.791006 7079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51232fccd98c461fb3d9016081f52ba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:10.791134 4354 tablet_service.cc:1467] Processing CreateTablet for tablet 681478951a8143f9959b78eec15efaf1 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=90e5baaad90e44bc8ebd55fcb5256a71]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:10.791213 6815 tablet_service.cc:1467] Processing CreateTablet for tablet 51232fccd98c461fb3d9016081f52ba5 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=90e5baaad90e44bc8ebd55fcb5256a71]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:10.791422 4354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 681478951a8143f9959b78eec15efaf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:10.791453 6815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51232fccd98c461fb3d9016081f52ba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:10.791462 4487 tablet_service.cc:1467] Processing CreateTablet for tablet 51232fccd98c461fb3d9016081f52ba5 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=90e5baaad90e44bc8ebd55fcb5256a71]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:10.791716 4487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51232fccd98c461fb3d9016081f52ba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:10.793884 6947 tablet_service.cc:1467] Processing CreateTablet for tablet 681478951a8143f9959b78eec15efaf1 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=90e5baaad90e44bc8ebd55fcb5256a71]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:10.793864 6681 tablet_service.cc:1467] Processing CreateTablet for tablet 681478951a8143f9959b78eec15efaf1 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=90e5baaad90e44bc8ebd55fcb5256a71]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:10.794149 6681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 681478951a8143f9959b78eec15efaf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:10.794145 6947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 681478951a8143f9959b78eec15efaf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:10.794338 9380 tablet_bootstrap.cc:492] T 51232fccd98c461fb3d9016081f52ba5 P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:46:10.794638 9378 tablet_bootstrap.cc:492] T 51232fccd98c461fb3d9016081f52ba5 P 201d2b6485084e7dad2cbb722af94779: Bootstrap starting.
I20240502 20:46:10.795289 9380 tablet_bootstrap.cc:654] T 51232fccd98c461fb3d9016081f52ba5 P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:10.795483 9378 tablet_bootstrap.cc:654] T 51232fccd98c461fb3d9016081f52ba5 P 201d2b6485084e7dad2cbb722af94779: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:10.796594 9378 tablet_bootstrap.cc:492] T 51232fccd98c461fb3d9016081f52ba5 P 201d2b6485084e7dad2cbb722af94779: No bootstrap required, opened a new log
I20240502 20:46:10.796612 9262 tablet_bootstrap.cc:492] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:46:10.796681 9378 ts_tablet_manager.cc:1367] T 51232fccd98c461fb3d9016081f52ba5 P 201d2b6485084e7dad2cbb722af94779: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:10.797209 9378 raft_consensus.cc:348] T 51232fccd98c461fb3d9016081f52ba5 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:10.797323 9378 raft_consensus.cc:374] T 51232fccd98c461fb3d9016081f52ba5 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:10.797358 9378 raft_consensus.cc:729] T 51232fccd98c461fb3d9016081f52ba5 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Initialized, Role: FOLLOWER
I20240502 20:46:10.797480 9378 consensus_queue.cc:260] T 51232fccd98c461fb3d9016081f52ba5 P 201d2b6485084e7dad2cbb722af94779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:10.797781 9378 ts_tablet_manager.cc:1398] T 51232fccd98c461fb3d9016081f52ba5 P 201d2b6485084e7dad2cbb722af94779: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:10.798028 9263 tablet_bootstrap.cc:492] T 681478951a8143f9959b78eec15efaf1 P 7845c397230d49358ba2fad9db53f97c: Bootstrap starting.
I20240502 20:46:10.799090 9263 tablet_bootstrap.cc:654] T 681478951a8143f9959b78eec15efaf1 P 7845c397230d49358ba2fad9db53f97c: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:10.799326 9380 tablet_bootstrap.cc:492] T 51232fccd98c461fb3d9016081f52ba5 P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:46:10.799423 9380 ts_tablet_manager.cc:1367] T 51232fccd98c461fb3d9016081f52ba5 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 20:46:10.799934 9380 raft_consensus.cc:348] T 51232fccd98c461fb3d9016081f52ba5 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:10.800052 9380 raft_consensus.cc:374] T 51232fccd98c461fb3d9016081f52ba5 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:10.800083 9380 raft_consensus.cc:729] T 51232fccd98c461fb3d9016081f52ba5 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:46:10.800199 9380 consensus_queue.cc:260] T 51232fccd98c461fb3d9016081f52ba5 P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:10.800258 4487 tablet_service.cc:1467] Processing CreateTablet for tablet 6c8f977981c54fa1bf5d173e83608146 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=90e5baaad90e44bc8ebd55fcb5256a71]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:10.800356 9262 tablet_bootstrap.cc:654] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:10.800499 4487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c8f977981c54fa1bf5d173e83608146. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:10.800753 7079 tablet_service.cc:1467] Processing CreateTablet for tablet 6c8f977981c54fa1bf5d173e83608146 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=90e5baaad90e44bc8ebd55fcb5256a71]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:10.801292 4750 tablet_service.cc:1467] Processing CreateTablet for tablet 6c8f977981c54fa1bf5d173e83608146 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=90e5baaad90e44bc8ebd55fcb5256a71]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:10.801560 4750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c8f977981c54fa1bf5d173e83608146. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:10.803483 9263 tablet_bootstrap.cc:492] T 681478951a8143f9959b78eec15efaf1 P 7845c397230d49358ba2fad9db53f97c: No bootstrap required, opened a new log
I20240502 20:46:10.803591 9263 ts_tablet_manager.cc:1367] T 681478951a8143f9959b78eec15efaf1 P 7845c397230d49358ba2fad9db53f97c: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 20:46:10.804102 9263 raft_consensus.cc:348] T 681478951a8143f9959b78eec15efaf1 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:10.804208 9263 raft_consensus.cc:374] T 681478951a8143f9959b78eec15efaf1 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:10.804237 9263 raft_consensus.cc:729] T 681478951a8143f9959b78eec15efaf1 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Initialized, Role: FOLLOWER
I20240502 20:46:10.804354 9263 consensus_queue.cc:260] T 681478951a8143f9959b78eec15efaf1 P 7845c397230d49358ba2fad9db53f97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:10.804497 4637 consensus_queue.cc:237] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } }
I20240502 20:46:10.804637 9263 ts_tablet_manager.cc:1398] T 681478951a8143f9959b78eec15efaf1 P 7845c397230d49358ba2fad9db53f97c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:10.805619 9379 tablet_bootstrap.cc:492] T 51232fccd98c461fb3d9016081f52ba5 P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap starting.
I20240502 20:46:10.805933 9262 tablet_bootstrap.cc:492] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:46:10.806031 9262 ts_tablet_manager.cc:1367] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 20:46:10.806538 4374 raft_consensus.cc:1261] T a7ea68e8581a4cd5a6212abc96dfbcde P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:10.806497 9262 raft_consensus.cc:348] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:10.806593 9379 tablet_bootstrap.cc:654] T 51232fccd98c461fb3d9016081f52ba5 P a4bf1df7fa6b4948be502c182bfbb833: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:10.806622 9262 raft_consensus.cc:374] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:10.806654 9262 raft_consensus.cc:729] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:46:10.806785 9262 consensus_queue.cc:260] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:10.806888 4768 raft_consensus.cc:1261] T a7ea68e8581a4cd5a6212abc96dfbcde P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:10.807116 9262 ts_tablet_manager.cc:1398] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:10.807299 7017 consensus_queue.cc:1035] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:10.807435 9385 tablet_bootstrap.cc:492] T 681478951a8143f9959b78eec15efaf1 P a47e5c56f13843018ac584914a8d4707: Bootstrap starting.
W20240502 20:46:10.808216 4572 consensus_peers.cc:485] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a7ea68e8581a4cd5a6212abc96dfbcde. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:10.808688 9385 tablet_bootstrap.cc:654] T 681478951a8143f9959b78eec15efaf1 P a47e5c56f13843018ac584914a8d4707: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:10.809247 7017 consensus_queue.cc:1035] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:10.809706 7079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c8f977981c54fa1bf5d173e83608146. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:10.810022 9379 tablet_bootstrap.cc:492] T 51232fccd98c461fb3d9016081f52ba5 P a4bf1df7fa6b4948be502c182bfbb833: No bootstrap required, opened a new log
I20240502 20:46:10.810093 9379 ts_tablet_manager.cc:1367] T 51232fccd98c461fb3d9016081f52ba5 P a4bf1df7fa6b4948be502c182bfbb833: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 20:46:10.810287 9385 tablet_bootstrap.cc:492] T 681478951a8143f9959b78eec15efaf1 P a47e5c56f13843018ac584914a8d4707: No bootstrap required, opened a new log
I20240502 20:46:10.810374 9385 ts_tablet_manager.cc:1367] T 681478951a8143f9959b78eec15efaf1 P a47e5c56f13843018ac584914a8d4707: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:46:10.810412 9379 raft_consensus.cc:348] T 51232fccd98c461fb3d9016081f52ba5 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:10.810488 9379 raft_consensus.cc:374] T 51232fccd98c461fb3d9016081f52ba5 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:10.810508 9379 raft_consensus.cc:729] T 51232fccd98c461fb3d9016081f52ba5 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Initialized, Role: FOLLOWER
I20240502 20:46:10.810827 9385 raft_consensus.cc:348] T 681478951a8143f9959b78eec15efaf1 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:10.811126 9379 consensus_queue.cc:260] T 51232fccd98c461fb3d9016081f52ba5 P a4bf1df7fa6b4948be502c182bfbb833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:10.810937 9385 raft_consensus.cc:374] T 681478951a8143f9959b78eec15efaf1 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:10.811699 9385 raft_consensus.cc:729] T 681478951a8143f9959b78eec15efaf1 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Initialized, Role: FOLLOWER
I20240502 20:46:10.811802 9379 ts_tablet_manager.cc:1398] T 51232fccd98c461fb3d9016081f52ba5 P a4bf1df7fa6b4948be502c182bfbb833: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:10.811897 9385 consensus_queue.cc:260] T 681478951a8143f9959b78eec15efaf1 P a47e5c56f13843018ac584914a8d4707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:10.812203 9385 ts_tablet_manager.cc:1398] T 681478951a8143f9959b78eec15efaf1 P a47e5c56f13843018ac584914a8d4707: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:10.812292 9378 tablet_bootstrap.cc:492] T 6c8f977981c54fa1bf5d173e83608146 P 201d2b6485084e7dad2cbb722af94779: Bootstrap starting.
I20240502 20:46:10.813323 9378 tablet_bootstrap.cc:654] T 6c8f977981c54fa1bf5d173e83608146 P 201d2b6485084e7dad2cbb722af94779: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:10.815268 9378 tablet_bootstrap.cc:492] T 6c8f977981c54fa1bf5d173e83608146 P 201d2b6485084e7dad2cbb722af94779: No bootstrap required, opened a new log
I20240502 20:46:10.815371 9378 ts_tablet_manager.cc:1367] T 6c8f977981c54fa1bf5d173e83608146 P 201d2b6485084e7dad2cbb722af94779: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:46:10.800499 9380 ts_tablet_manager.cc:1398] T 51232fccd98c461fb3d9016081f52ba5 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:10.815858 9378 raft_consensus.cc:348] T 6c8f977981c54fa1bf5d173e83608146 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:10.815981 9378 raft_consensus.cc:374] T 6c8f977981c54fa1bf5d173e83608146 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:10.816011 9378 raft_consensus.cc:729] T 6c8f977981c54fa1bf5d173e83608146 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Initialized, Role: FOLLOWER
I20240502 20:46:10.816140 9378 consensus_queue.cc:260] T 6c8f977981c54fa1bf5d173e83608146 P 201d2b6485084e7dad2cbb722af94779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:10.816195 9380 tablet_bootstrap.cc:492] T 6c8f977981c54fa1bf5d173e83608146 P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:46:10.816453 9378 ts_tablet_manager.cc:1398] T 6c8f977981c54fa1bf5d173e83608146 P 201d2b6485084e7dad2cbb722af94779: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:10.817338 9380 tablet_bootstrap.cc:654] T 6c8f977981c54fa1bf5d173e83608146 P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:10.818193 9041 raft_consensus.cc:2932] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } } }
I20240502 20:46:10.818673 9386 tablet_bootstrap.cc:492] T 6c8f977981c54fa1bf5d173e83608146 P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:46:10.818699 4768 raft_consensus.cc:2932] T a7ea68e8581a4cd5a6212abc96dfbcde P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } } }
I20240502 20:46:10.819581 9386 tablet_bootstrap.cc:654] T 6c8f977981c54fa1bf5d173e83608146 P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:10.821635 9380 tablet_bootstrap.cc:492] T 6c8f977981c54fa1bf5d173e83608146 P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:46:10.821731 9380 ts_tablet_manager.cc:1367] T 6c8f977981c54fa1bf5d173e83608146 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 20:46:10.821583 4230 catalog_manager.cc:5484] T a7ea68e8581a4cd5a6212abc96dfbcde P ed47c023452a497192f35d2f617bf9bd reported cstate change: config changed from index -1 to 4, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } } }
I20240502 20:46:10.822206 9380 raft_consensus.cc:348] T 6c8f977981c54fa1bf5d173e83608146 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:10.822304 9380 raft_consensus.cc:374] T 6c8f977981c54fa1bf5d173e83608146 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:10.822325 9380 raft_consensus.cc:729] T 6c8f977981c54fa1bf5d173e83608146 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:46:10.822221 4377 raft_consensus.cc:2932] T a7ea68e8581a4cd5a6212abc96dfbcde P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } } }
I20240502 20:46:10.822422 9380 consensus_queue.cc:260] T 6c8f977981c54fa1bf5d173e83608146 P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:10.822697 9380 ts_tablet_manager.cc:1398] T 6c8f977981c54fa1bf5d173e83608146 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:10.825181 9386 tablet_bootstrap.cc:492] T 6c8f977981c54fa1bf5d173e83608146 P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:46:10.825361 9386 ts_tablet_manager.cc:1367] T 6c8f977981c54fa1bf5d173e83608146 P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 20:46:10.825894 9386 raft_consensus.cc:348] T 6c8f977981c54fa1bf5d173e83608146 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:10.826097 9386 raft_consensus.cc:374] T 6c8f977981c54fa1bf5d173e83608146 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:10.826182 9386 raft_consensus.cc:729] T 6c8f977981c54fa1bf5d173e83608146 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:46:10.826350 9386 consensus_queue.cc:260] T 6c8f977981c54fa1bf5d173e83608146 P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:10.826632 9386 ts_tablet_manager.cc:1398] T 6c8f977981c54fa1bf5d173e83608146 P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:10.827922 7967 raft_consensus.cc:482] T 6c8f977981c54fa1bf5d173e83608146 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:10.828027 7967 raft_consensus.cc:504] T 6c8f977981c54fa1bf5d173e83608146 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:10.828467 7967 leader_election.cc:288] T 6c8f977981c54fa1bf5d173e83608146 P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
I20240502 20:46:10.828852 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8f977981c54fa1bf5d173e83608146" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd" is_pre_election: true
I20240502 20:46:10.829007 4768 raft_consensus.cc:2454] T 6c8f977981c54fa1bf5d173e83608146 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 0.
I20240502 20:46:10.829253 7099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8f977981c54fa1bf5d173e83608146" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201d2b6485084e7dad2cbb722af94779" is_pre_election: true
I20240502 20:46:10.829396 7099 raft_consensus.cc:2454] T 6c8f977981c54fa1bf5d173e83608146 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 0.
I20240502 20:46:10.829746 4441 leader_election.cc:302] T 6c8f977981c54fa1bf5d173e83608146 P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201d2b6485084e7dad2cbb722af94779, c3d26eafd51547acb780d4aac5ebe7cc; no voters:
I20240502 20:46:10.829954 7967 raft_consensus.cc:2781] T 6c8f977981c54fa1bf5d173e83608146 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:10.830003 7967 raft_consensus.cc:482] T 6c8f977981c54fa1bf5d173e83608146 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:10.830040 7967 raft_consensus.cc:3037] T 6c8f977981c54fa1bf5d173e83608146 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:10.831111 7967 raft_consensus.cc:504] T 6c8f977981c54fa1bf5d173e83608146 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:10.831442 7967 leader_election.cc:288] T 6c8f977981c54fa1bf5d173e83608146 P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 election: Requested vote from peers ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
I20240502 20:46:10.832410 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8f977981c54fa1bf5d173e83608146" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd"
I20240502 20:46:10.832530 4768 raft_consensus.cc:3037] T 6c8f977981c54fa1bf5d173e83608146 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:10.833362 7099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8f977981c54fa1bf5d173e83608146" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201d2b6485084e7dad2cbb722af94779"
I20240502 20:46:10.833483 7099 raft_consensus.cc:3037] T 6c8f977981c54fa1bf5d173e83608146 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:10.833621 4768 raft_consensus.cc:2454] T 6c8f977981c54fa1bf5d173e83608146 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 1.
I20240502 20:46:10.833920 4444 leader_election.cc:302] T 6c8f977981c54fa1bf5d173e83608146 P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3d26eafd51547acb780d4aac5ebe7cc, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:46:10.834170 7967 raft_consensus.cc:2781] T 6c8f977981c54fa1bf5d173e83608146 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:10.834270 7967 raft_consensus.cc:686] T 6c8f977981c54fa1bf5d173e83608146 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Becoming Leader. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Running, Role: LEADER
I20240502 20:46:10.834419 7967 consensus_queue.cc:237] T 6c8f977981c54fa1bf5d173e83608146 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:10.834532 7099 raft_consensus.cc:2454] T 6c8f977981c54fa1bf5d173e83608146 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 1.
I20240502 20:46:10.836477 4230 catalog_manager.cc:5484] T 6c8f977981c54fa1bf5d173e83608146 P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: term changed from 0 to 1, leader changed from <none> to c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66). New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } }
I20240502 20:46:10.837908 6834 consensus_queue.cc:237] T 52e302882311488ab1cac90ad03fe270 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } }
I20240502 20:46:10.839458 7099 raft_consensus.cc:1261] T 52e302882311488ab1cac90ad03fe270 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Refusing update from remote peer a4bf1df7fa6b4948be502c182bfbb833: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 20:46:10.839696 6770 consensus_peers.cc:485] T 52e302882311488ab1cac90ad03fe270 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097): Couldn't send request to peer c3d26eafd51547acb780d4aac5ebe7cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 52e302882311488ab1cac90ad03fe270. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:10.840023 6967 raft_consensus.cc:1261] T 52e302882311488ab1cac90ad03fe270 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Refusing update from remote peer a4bf1df7fa6b4948be502c182bfbb833: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:10.840248 9033 consensus_queue.cc:1035] T 52e302882311488ab1cac90ad03fe270 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:46:10.840420 8364 consensus_queue.cc:1035] T 52e302882311488ab1cac90ad03fe270 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:46:10.842268 8364 raft_consensus.cc:2932] T 52e302882311488ab1cac90ad03fe270 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } } }
I20240502 20:46:10.842703 6967 raft_consensus.cc:2932] T 52e302882311488ab1cac90ad03fe270 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } } }
I20240502 20:46:10.843201 7099 raft_consensus.cc:2932] T 52e302882311488ab1cac90ad03fe270 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } } }
I20240502 20:46:10.845219 4230 catalog_manager.cc:5484] T 52e302882311488ab1cac90ad03fe270 P a47e5c56f13843018ac584914a8d4707 reported cstate change: config changed from index -1 to 2, NON_VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) added. New cstate: current_term: 1 leader_uuid: "a4bf1df7fa6b4948be502c182bfbb833" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } } }
I20240502 20:46:10.850023 8460 raft_consensus.cc:482] T 51232fccd98c461fb3d9016081f52ba5 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:10.850109 8460 raft_consensus.cc:504] T 51232fccd98c461fb3d9016081f52ba5 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:10.850589 8460 leader_election.cc:288] T 51232fccd98c461fb3d9016081f52ba5 P 201d2b6485084e7dad2cbb722af94779 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097), a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867)
I20240502 20:46:10.850908 4507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51232fccd98c461fb3d9016081f52ba5" candidate_uuid: "201d2b6485084e7dad2cbb722af94779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" is_pre_election: true
I20240502 20:46:10.851054 4507 raft_consensus.cc:2454] T 51232fccd98c461fb3d9016081f52ba5 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 201d2b6485084e7dad2cbb722af94779 in term 0.
I20240502 20:46:10.851044 6835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51232fccd98c461fb3d9016081f52ba5" candidate_uuid: "201d2b6485084e7dad2cbb722af94779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833" is_pre_election: true
I20240502 20:46:10.851147 6835 raft_consensus.cc:2454] T 51232fccd98c461fb3d9016081f52ba5 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 201d2b6485084e7dad2cbb722af94779 in term 0.
I20240502 20:46:10.851588 7034 leader_election.cc:302] T 51232fccd98c461fb3d9016081f52ba5 P 201d2b6485084e7dad2cbb722af94779 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201d2b6485084e7dad2cbb722af94779, c3d26eafd51547acb780d4aac5ebe7cc; no voters:
I20240502 20:46:10.852524 8460 raft_consensus.cc:2781] T 51232fccd98c461fb3d9016081f52ba5 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:10.852612 8460 raft_consensus.cc:482] T 51232fccd98c461fb3d9016081f52ba5 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:10.852655 8460 raft_consensus.cc:3037] T 51232fccd98c461fb3d9016081f52ba5 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:10.853708 8460 raft_consensus.cc:504] T 51232fccd98c461fb3d9016081f52ba5 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:10.854038 8460 leader_election.cc:288] T 51232fccd98c461fb3d9016081f52ba5 P 201d2b6485084e7dad2cbb722af94779 [CANDIDATE]: Term 1 election: Requested vote from peers c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097), a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867)
I20240502 20:46:10.854526 6835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51232fccd98c461fb3d9016081f52ba5" candidate_uuid: "201d2b6485084e7dad2cbb722af94779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833"
I20240502 20:46:10.854631 6835 raft_consensus.cc:3037] T 51232fccd98c461fb3d9016081f52ba5 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:10.854651 4507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51232fccd98c461fb3d9016081f52ba5" candidate_uuid: "201d2b6485084e7dad2cbb722af94779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc"
I20240502 20:46:10.854754 4507 raft_consensus.cc:3037] T 51232fccd98c461fb3d9016081f52ba5 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:10.854708 6967 consensus_queue.cc:237] T 8602022a45e1482ebf8dab9930ce8209 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } }
I20240502 20:46:10.855589 6835 raft_consensus.cc:2454] T 51232fccd98c461fb3d9016081f52ba5 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 201d2b6485084e7dad2cbb722af94779 in term 1.
I20240502 20:46:10.855729 4507 raft_consensus.cc:2454] T 51232fccd98c461fb3d9016081f52ba5 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 201d2b6485084e7dad2cbb722af94779 in term 1.
I20240502 20:46:10.856462 6703 raft_consensus.cc:1261] T 8602022a45e1482ebf8dab9930ce8209 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Refusing update from remote peer a47e5c56f13843018ac584914a8d4707: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 20:46:10.856534 6900 consensus_peers.cc:485] T 8602022a45e1482ebf8dab9930ce8209 P a47e5c56f13843018ac584914a8d4707 -> Peer 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141): Couldn't send request to peer 923cfab926e7443f8ffd7b21ad0a9d91. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8602022a45e1482ebf8dab9930ce8209. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:10.856827 8304 consensus_queue.cc:1035] T 8602022a45e1482ebf8dab9930ce8209 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:46:10.856909 7033 leader_election.cc:302] T 51232fccd98c461fb3d9016081f52ba5 P 201d2b6485084e7dad2cbb722af94779 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201d2b6485084e7dad2cbb722af94779, a4bf1df7fa6b4948be502c182bfbb833; no voters:
I20240502 20:46:10.857178 6835 raft_consensus.cc:1261] T 8602022a45e1482ebf8dab9930ce8209 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Refusing update from remote peer a47e5c56f13843018ac584914a8d4707: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:10.857538 8304 consensus_queue.cc:1035] T 8602022a45e1482ebf8dab9930ce8209 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:46:10.857990 4228 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 20:46:10.858690 8460 raft_consensus.cc:2781] T 51232fccd98c461fb3d9016081f52ba5 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:10.858861 8460 raft_consensus.cc:686] T 51232fccd98c461fb3d9016081f52ba5 P 201d2b6485084e7dad2cbb722af94779 [term 1 LEADER]: Becoming Leader. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Running, Role: LEADER
I20240502 20:46:10.858999 8460 consensus_queue.cc:237] T 51232fccd98c461fb3d9016081f52ba5 P 201d2b6485084e7dad2cbb722af94779 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:10.860780 4233 catalog_manager.cc:5484] T 51232fccd98c461fb3d9016081f52ba5 P 201d2b6485084e7dad2cbb722af94779 reported cstate change: term changed from 0 to 1, leader changed from <none> to 201d2b6485084e7dad2cbb722af94779 (127.30.32.72). New cstate: current_term: 1 leader_uuid: "201d2b6485084e7dad2cbb722af94779" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:10.862030 8304 raft_consensus.cc:2932] T 8602022a45e1482ebf8dab9930ce8209 P a47e5c56f13843018ac584914a8d4707 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } } }
I20240502 20:46:10.864706 6835 raft_consensus.cc:2932] T 8602022a45e1482ebf8dab9930ce8209 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } } }
I20240502 20:46:10.864884 9357 tablet.cc:1722] T 8eebae62c8054dc48e485b7e45daeffa P c92e6e46782849479c8a0c6d82276fb2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 20:46:10.864914 4228 catalog_manager.cc:5484] T 8602022a45e1482ebf8dab9930ce8209 P a47e5c56f13843018ac584914a8d4707 reported cstate change: config changed from index -1 to 2, NON_VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) added. New cstate: current_term: 1 leader_uuid: "a47e5c56f13843018ac584914a8d4707" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:10.865253 9357 tablet.cc:1620] T 8eebae62c8054dc48e485b7e45daeffa P c92e6e46782849479c8a0c6d82276fb2: MemRowSet was empty: no flush needed.
I20240502 20:46:10.867003 9364 tablet.cc:1722] T 8eebae62c8054dc48e485b7e45daeffa P 7845c397230d49358ba2fad9db53f97c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 20:46:10.867142 6702 raft_consensus.cc:2932] T 8602022a45e1482ebf8dab9930ce8209 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } } }
I20240502 20:46:10.867297 9364 tablet.cc:1620] T 8eebae62c8054dc48e485b7e45daeffa P 7845c397230d49358ba2fad9db53f97c: MemRowSet was empty: no flush needed.
I20240502 20:46:10.871766 9363 tablet.cc:1722] T 8eebae62c8054dc48e485b7e45daeffa P 923cfab926e7443f8ffd7b21ad0a9d91: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 20:46:10.872067 9363 tablet.cc:1620] T 8eebae62c8054dc48e485b7e45daeffa P 923cfab926e7443f8ffd7b21ad0a9d91: MemRowSet was empty: no flush needed.
I20240502 20:46:10.872138 4227 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: rebalancer_dynamic_table [id=83a915ef71d94c1da0ade39f98e82b29] alter complete (version 2)
I20240502 20:46:10.972446 4768 consensus_queue.cc:237] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } }
I20240502 20:46:10.974031 4637 raft_consensus.cc:1261] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:10.974341 4228 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 20:46:10.974453 9125 consensus_queue.cc:1035] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:10.974992 6702 raft_consensus.cc:1261] T d37fab152a8a44868d3160ec3057bb98 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:10.975529 9125 consensus_queue.cc:1035] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 20:46:10.976172 4706 consensus_peers.cc:485] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d37fab152a8a44868d3160ec3057bb98. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:10.977221 9343 raft_consensus.cc:2932] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } } }
I20240502 20:46:10.978142 4637 raft_consensus.cc:2932] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } } }
I20240502 20:46:10.980732 9357 tablet.cc:1722] T 8eebae62c8054dc48e485b7e45daeffa P c92e6e46782849479c8a0c6d82276fb2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 20:46:10.981578 4227 catalog_manager.cc:5484] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd reported cstate change: config changed from index 6 to 7, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:10.982956 9363 tablet.cc:1722] T 8eebae62c8054dc48e485b7e45daeffa P 923cfab926e7443f8ffd7b21ad0a9d91: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 20:46:10.983743 9364 tablet.cc:1722] T 8eebae62c8054dc48e485b7e45daeffa P 7845c397230d49358ba2fad9db53f97c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 20:46:10.983752 6703 raft_consensus.cc:2932] T d37fab152a8a44868d3160ec3057bb98 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } } }
I20240502 20:46:10.988016 4230 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: rebalancer_dynamic_table_new_name [id=83a915ef71d94c1da0ade39f98e82b29] alter complete (version 3)
I20240502 20:46:10.997148 9514 ts_tablet_manager.cc:902] T 02302fc79dc04b0396041c600c2f4bef P 7845c397230d49358ba2fad9db53f97c: Initiating tablet copy from peer 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:46:10.997715 9514 tablet_copy_client.cc:308] T 02302fc79dc04b0396041c600c2f4bef P 7845c397230d49358ba2fad9db53f97c: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.67:42141
I20240502 20:46:10.998168 4658 tablet_copy_service.cc:140] P 923cfab926e7443f8ffd7b21ad0a9d91: Received BeginTabletCopySession request for tablet 02302fc79dc04b0396041c600c2f4bef from peer 7845c397230d49358ba2fad9db53f97c ({username='slave'} at 127.30.32.69:34201)
I20240502 20:46:10.998282 4658 tablet_copy_service.cc:161] P 923cfab926e7443f8ffd7b21ad0a9d91: Beginning new tablet copy session on tablet 02302fc79dc04b0396041c600c2f4bef from peer 7845c397230d49358ba2fad9db53f97c at {username='slave'} at 127.30.32.69:34201: session id = 7845c397230d49358ba2fad9db53f97c-02302fc79dc04b0396041c600c2f4bef
I20240502 20:46:10.999334 4658 tablet_copy_source_session.cc:205] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:11.000233 9514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02302fc79dc04b0396041c600c2f4bef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:11.002637 9514 tablet_copy_client.cc:783] T 02302fc79dc04b0396041c600c2f4bef P 7845c397230d49358ba2fad9db53f97c: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:11.002810 9514 tablet_copy_client.cc:647] T 02302fc79dc04b0396041c600c2f4bef P 7845c397230d49358ba2fad9db53f97c: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:11.004101 9514 tablet_copy_client.cc:523] T 02302fc79dc04b0396041c600c2f4bef P 7845c397230d49358ba2fad9db53f97c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:11.005308 9514 tablet_bootstrap.cc:492] T 02302fc79dc04b0396041c600c2f4bef P 7845c397230d49358ba2fad9db53f97c: Bootstrap starting.
I20240502 20:46:11.022997 9535 ts_tablet_manager.cc:902] T 41f84d5db2034af591b56e109be67934 P 201d2b6485084e7dad2cbb722af94779: Initiating tablet copy from peer c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097)
I20240502 20:46:11.024087 9535 tablet_copy_client.cc:308] T 41f84d5db2034af591b56e109be67934 P 201d2b6485084e7dad2cbb722af94779: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.66:42097
I20240502 20:46:11.025274 4528 tablet_copy_service.cc:140] P c3d26eafd51547acb780d4aac5ebe7cc: Received BeginTabletCopySession request for tablet 41f84d5db2034af591b56e109be67934 from peer 201d2b6485084e7dad2cbb722af94779 ({username='slave'} at 127.30.32.72:54159)
I20240502 20:46:11.025385 4528 tablet_copy_service.cc:161] P c3d26eafd51547acb780d4aac5ebe7cc: Beginning new tablet copy session on tablet 41f84d5db2034af591b56e109be67934 from peer 201d2b6485084e7dad2cbb722af94779 at {username='slave'} at 127.30.32.72:54159: session id = 201d2b6485084e7dad2cbb722af94779-41f84d5db2034af591b56e109be67934
I20240502 20:46:11.026317 4528 tablet_copy_source_session.cc:205] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:11.027664 9535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41f84d5db2034af591b56e109be67934. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:11.029909 9535 tablet_copy_client.cc:783] T 41f84d5db2034af591b56e109be67934 P 201d2b6485084e7dad2cbb722af94779: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:11.030061 9535 tablet_copy_client.cc:647] T 41f84d5db2034af591b56e109be67934 P 201d2b6485084e7dad2cbb722af94779: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:11.030730 9514 tablet_bootstrap.cc:492] T 02302fc79dc04b0396041c600c2f4bef P 7845c397230d49358ba2fad9db53f97c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:11.031162 9514 tablet_bootstrap.cc:492] T 02302fc79dc04b0396041c600c2f4bef P 7845c397230d49358ba2fad9db53f97c: Bootstrap complete.
I20240502 20:46:11.031203 9535 tablet_copy_client.cc:523] T 41f84d5db2034af591b56e109be67934 P 201d2b6485084e7dad2cbb722af94779: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:11.031512 9514 ts_tablet_manager.cc:1367] T 02302fc79dc04b0396041c600c2f4bef P 7845c397230d49358ba2fad9db53f97c: Time spent bootstrapping tablet: real 0.026s user 0.022s sys 0.004s
I20240502 20:46:11.031992 9514 raft_consensus.cc:348] T 02302fc79dc04b0396041c600c2f4bef P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } }
I20240502 20:46:11.032230 9514 raft_consensus.cc:729] T 02302fc79dc04b0396041c600c2f4bef P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Initialized, Role: LEARNER
I20240502 20:46:11.032418 9514 consensus_queue.cc:260] T 02302fc79dc04b0396041c600c2f4bef P 7845c397230d49358ba2fad9db53f97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } }
I20240502 20:46:11.032835 9514 ts_tablet_manager.cc:1398] T 02302fc79dc04b0396041c600c2f4bef P 7845c397230d49358ba2fad9db53f97c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:11.032634 9535 tablet_bootstrap.cc:492] T 41f84d5db2034af591b56e109be67934 P 201d2b6485084e7dad2cbb722af94779: Bootstrap starting.
I20240502 20:46:11.033799 4658 tablet_copy_service.cc:342] P 923cfab926e7443f8ffd7b21ad0a9d91: Request end of tablet copy session 7845c397230d49358ba2fad9db53f97c-02302fc79dc04b0396041c600c2f4bef received from {username='slave'} at 127.30.32.69:34201
I20240502 20:46:11.033883 4658 tablet_copy_service.cc:433] P 923cfab926e7443f8ffd7b21ad0a9d91: ending tablet copy session 7845c397230d49358ba2fad9db53f97c-02302fc79dc04b0396041c600c2f4bef on tablet 02302fc79dc04b0396041c600c2f4bef with peer 7845c397230d49358ba2fad9db53f97c
I20240502 20:46:11.056882 9556 ts_tablet_manager.cc:902] T 00eb91e624554baf918e96b098019be8 P a47e5c56f13843018ac584914a8d4707: Initiating tablet copy from peer 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:46:11.057452 9556 tablet_copy_client.cc:308] T 00eb91e624554baf918e96b098019be8 P a47e5c56f13843018ac584914a8d4707: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.67:42141
I20240502 20:46:11.057889 4658 tablet_copy_service.cc:140] P 923cfab926e7443f8ffd7b21ad0a9d91: Received BeginTabletCopySession request for tablet 00eb91e624554baf918e96b098019be8 from peer a47e5c56f13843018ac584914a8d4707 ({username='slave'} at 127.30.32.71:59213)
I20240502 20:46:11.057993 4658 tablet_copy_service.cc:161] P 923cfab926e7443f8ffd7b21ad0a9d91: Beginning new tablet copy session on tablet 00eb91e624554baf918e96b098019be8 from peer a47e5c56f13843018ac584914a8d4707 at {username='slave'} at 127.30.32.71:59213: session id = a47e5c56f13843018ac584914a8d4707-00eb91e624554baf918e96b098019be8
I20240502 20:46:11.058791 4658 tablet_copy_source_session.cc:205] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:11.060768 9556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00eb91e624554baf918e96b098019be8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:11.063119 9556 tablet_copy_client.cc:783] T 00eb91e624554baf918e96b098019be8 P a47e5c56f13843018ac584914a8d4707: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:11.063268 9556 tablet_copy_client.cc:647] T 00eb91e624554baf918e96b098019be8 P a47e5c56f13843018ac584914a8d4707: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:11.064824 9556 tablet_copy_client.cc:523] T 00eb91e624554baf918e96b098019be8 P a47e5c56f13843018ac584914a8d4707: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:11.067803 9556 tablet_bootstrap.cc:492] T 00eb91e624554baf918e96b098019be8 P a47e5c56f13843018ac584914a8d4707: Bootstrap starting.
I20240502 20:46:11.069470 9535 tablet_bootstrap.cc:492] T 41f84d5db2034af591b56e109be67934 P 201d2b6485084e7dad2cbb722af94779: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:11.073316 9535 tablet_bootstrap.cc:492] T 41f84d5db2034af591b56e109be67934 P 201d2b6485084e7dad2cbb722af94779: Bootstrap complete.
I20240502 20:46:11.073521 9535 ts_tablet_manager.cc:1367] T 41f84d5db2034af591b56e109be67934 P 201d2b6485084e7dad2cbb722af94779: Time spent bootstrapping tablet: real 0.041s user 0.026s sys 0.000s
I20240502 20:46:11.073964 9535 raft_consensus.cc:348] T 41f84d5db2034af591b56e109be67934 P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } }
I20240502 20:46:11.074086 9535 raft_consensus.cc:729] T 41f84d5db2034af591b56e109be67934 P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Initialized, Role: LEARNER
I20240502 20:46:11.074184 9535 consensus_queue.cc:260] T 41f84d5db2034af591b56e109be67934 P 201d2b6485084e7dad2cbb722af94779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } }
I20240502 20:46:11.074451 9535 ts_tablet_manager.cc:1398] T 41f84d5db2034af591b56e109be67934 P 201d2b6485084e7dad2cbb722af94779: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:11.075424 4528 tablet_copy_service.cc:342] P c3d26eafd51547acb780d4aac5ebe7cc: Request end of tablet copy session 201d2b6485084e7dad2cbb722af94779-41f84d5db2034af591b56e109be67934 received from {username='slave'} at 127.30.32.72:54159
I20240502 20:46:11.075517 4528 tablet_copy_service.cc:433] P c3d26eafd51547acb780d4aac5ebe7cc: ending tablet copy session 201d2b6485084e7dad2cbb722af94779-41f84d5db2034af591b56e109be67934 on tablet 41f84d5db2034af591b56e109be67934 with peer 201d2b6485084e7dad2cbb722af94779
I20240502 20:46:11.093710 4231 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 20:46:11.093878 4231 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 20:46:11.093899 9556 tablet_bootstrap.cc:492] T 00eb91e624554baf918e96b098019be8 P a47e5c56f13843018ac584914a8d4707: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:11.094221 9556 tablet_bootstrap.cc:492] T 00eb91e624554baf918e96b098019be8 P a47e5c56f13843018ac584914a8d4707: Bootstrap complete.
I20240502 20:46:11.094444 9556 ts_tablet_manager.cc:1367] T 00eb91e624554baf918e96b098019be8 P a47e5c56f13843018ac584914a8d4707: Time spent bootstrapping tablet: real 0.027s user 0.019s sys 0.003s
I20240502 20:46:11.094874 9556 raft_consensus.cc:348] T 00eb91e624554baf918e96b098019be8 P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } }
I20240502 20:46:11.094980 9556 raft_consensus.cc:729] T 00eb91e624554baf918e96b098019be8 P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Initialized, Role: LEARNER
I20240502 20:46:11.095090 9556 consensus_queue.cc:260] T 00eb91e624554baf918e96b098019be8 P a47e5c56f13843018ac584914a8d4707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } }
I20240502 20:46:11.096014 4231 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: Sending DeleteTablet for 3 replicas of tablet 8eebae62c8054dc48e485b7e45daeffa
I20240502 20:46:11.096431 9556 ts_tablet_manager.cc:1398] T 00eb91e624554baf918e96b098019be8 P a47e5c56f13843018ac584914a8d4707: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:11.096904 6681 tablet_service.cc:1514] Processing DeleteTablet for tablet 8eebae62c8054dc48e485b7e45daeffa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:11 UTC) from {username='slave'} at 127.0.0.1:37506
I20240502 20:46:11.096927 4354 tablet_service.cc:1514] Processing DeleteTablet for tablet 8eebae62c8054dc48e485b7e45daeffa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:11 UTC) from {username='slave'} at 127.0.0.1:38306
I20240502 20:46:11.097153 7297 tablet_replica.cc:330] T 8eebae62c8054dc48e485b7e45daeffa P c92e6e46782849479c8a0c6d82276fb2: stopping tablet replica
I20240502 20:46:11.097224 4658 tablet_copy_service.cc:342] P 923cfab926e7443f8ffd7b21ad0a9d91: Request end of tablet copy session a47e5c56f13843018ac584914a8d4707-00eb91e624554baf918e96b098019be8 received from {username='slave'} at 127.30.32.71:59213
I20240502 20:46:11.097306 4658 tablet_copy_service.cc:433] P 923cfab926e7443f8ffd7b21ad0a9d91: ending tablet copy session a47e5c56f13843018ac584914a8d4707-00eb91e624554baf918e96b098019be8 on tablet 00eb91e624554baf918e96b098019be8 with peer a47e5c56f13843018ac584914a8d4707
I20240502 20:46:11.097318 7297 raft_consensus.cc:2229] T 8eebae62c8054dc48e485b7e45daeffa P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:11.097396 4618 tablet_service.cc:1514] Processing DeleteTablet for tablet 8eebae62c8054dc48e485b7e45daeffa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:11 UTC) from {username='slave'} at 127.0.0.1:57068
I20240502 20:46:11.097563 7297 raft_consensus.cc:2258] T 8eebae62c8054dc48e485b7e45daeffa P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:11.098227 7297 ts_tablet_manager.cc:1875] T 8eebae62c8054dc48e485b7e45daeffa P c92e6e46782849479c8a0c6d82276fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:11.099118 9587 tablet_replica.cc:330] T 8eebae62c8054dc48e485b7e45daeffa P 7845c397230d49358ba2fad9db53f97c: stopping tablet replica
I20240502 20:46:11.099243 9587 raft_consensus.cc:2229] T 8eebae62c8054dc48e485b7e45daeffa P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:11.099359 9587 raft_consensus.cc:2258] T 8eebae62c8054dc48e485b7e45daeffa P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:11.099582 9588 tablet_replica.cc:330] T 8eebae62c8054dc48e485b7e45daeffa P 923cfab926e7443f8ffd7b21ad0a9d91: stopping tablet replica
I20240502 20:46:11.100106 9588 raft_consensus.cc:2229] T 8eebae62c8054dc48e485b7e45daeffa P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:11.100220 9588 raft_consensus.cc:2258] T 8eebae62c8054dc48e485b7e45daeffa P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:11.100692 7297 ts_tablet_manager.cc:1888] T 8eebae62c8054dc48e485b7e45daeffa P c92e6e46782849479c8a0c6d82276fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:11.100764 7297 log.cc:1196] T 8eebae62c8054dc48e485b7e45daeffa P c92e6e46782849479c8a0c6d82276fb2: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal/wals/8eebae62c8054dc48e485b7e45daeffa
I20240502 20:46:11.101037 9588 ts_tablet_manager.cc:1875] T 8eebae62c8054dc48e485b7e45daeffa P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:11.101066 7297 ts_tablet_manager.cc:1909] T 8eebae62c8054dc48e485b7e45daeffa P c92e6e46782849479c8a0c6d82276fb2: Deleting consensus metadata
I20240502 20:46:11.101786 4222 catalog_manager.cc:4830] TS c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827): tablet 8eebae62c8054dc48e485b7e45daeffa (table rebalancer_dynamic_table_new_name [id=83a915ef71d94c1da0ade39f98e82b29]) successfully deleted
I20240502 20:46:11.102130 9587 ts_tablet_manager.cc:1875] T 8eebae62c8054dc48e485b7e45daeffa P 7845c397230d49358ba2fad9db53f97c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:11.104458 7937 raft_consensus.cc:1050] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2: attempting to promote NON_VOTER 923cfab926e7443f8ffd7b21ad0a9d91 to VOTER
I20240502 20:46:11.104899 7937 consensus_queue.cc:237] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } }
I20240502 20:46:11.105458 9587 ts_tablet_manager.cc:1888] T 8eebae62c8054dc48e485b7e45daeffa P 7845c397230d49358ba2fad9db53f97c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:11.105537 9587 log.cc:1196] T 8eebae62c8054dc48e485b7e45daeffa P 7845c397230d49358ba2fad9db53f97c: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/wal/wals/8eebae62c8054dc48e485b7e45daeffa
I20240502 20:46:11.105767 9587 ts_tablet_manager.cc:1909] T 8eebae62c8054dc48e485b7e45daeffa P 7845c397230d49358ba2fad9db53f97c: Deleting consensus metadata
I20240502 20:46:11.106345 4507 raft_consensus.cc:1261] T 15a2cbe8285a4938be7f628f7f8b734d P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:11.106474 4638 raft_consensus.cc:1261] T 15a2cbe8285a4938be7f628f7f8b734d P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEARNER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:11.106586 7099 raft_consensus.cc:1261] T 15a2cbe8285a4938be7f628f7f8b734d P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:11.106828 8918 consensus_queue.cc:1035] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:46:11.106997 7937 consensus_queue.cc:1035] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:46:11.107060 9588 ts_tablet_manager.cc:1888] T 8eebae62c8054dc48e485b7e45daeffa P 923cfab926e7443f8ffd7b21ad0a9d91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:11.107159 9588 log.cc:1196] T 8eebae62c8054dc48e485b7e45daeffa P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal/wals/8eebae62c8054dc48e485b7e45daeffa
I20240502 20:46:11.107431 8185 consensus_queue.cc:1035] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:46:11.108371 4221 catalog_manager.cc:4830] TS 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): tablet 8eebae62c8054dc48e485b7e45daeffa (table rebalancer_dynamic_table_new_name [id=83a915ef71d94c1da0ade39f98e82b29]) successfully deleted
I20240502 20:46:11.109092 9588 ts_tablet_manager.cc:1909] T 8eebae62c8054dc48e485b7e45daeffa P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting consensus metadata
I20240502 20:46:11.109871 4219 catalog_manager.cc:4830] TS 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141): tablet 8eebae62c8054dc48e485b7e45daeffa (table rebalancer_dynamic_table_new_name [id=83a915ef71d94c1da0ade39f98e82b29]) successfully deleted
I20240502 20:46:11.110364 8918 raft_consensus.cc:2932] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:11.111519 4638 raft_consensus.cc:2932] T 15a2cbe8285a4938be7f628f7f8b734d P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:11.113109 7099 raft_consensus.cc:2932] T 15a2cbe8285a4938be7f628f7f8b734d P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:11.113121 4507 raft_consensus.cc:2932] T 15a2cbe8285a4938be7f628f7f8b734d P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:11.115190 4228 catalog_manager.cc:5484] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2 reported cstate change: config changed from index 2 to 3, 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c92e6e46782849479c8a0c6d82276fb2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:11.117347 9156 raft_consensus.cc:482] T 681478951a8143f9959b78eec15efaf1 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:11.117462 9156 raft_consensus.cc:504] T 681478951a8143f9959b78eec15efaf1 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:11.117929 9156 leader_election.cc:288] T 681478951a8143f9959b78eec15efaf1 P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:46:11.118580 4377 consensus_queue.cc:237] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } }
I20240502 20:46:11.119786 4377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681478951a8143f9959b78eec15efaf1" candidate_uuid: "a47e5c56f13843018ac584914a8d4707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2" is_pre_election: true
I20240502 20:46:11.119859 6703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681478951a8143f9959b78eec15efaf1" candidate_uuid: "a47e5c56f13843018ac584914a8d4707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7845c397230d49358ba2fad9db53f97c" is_pre_election: true
I20240502 20:46:11.119912 4377 raft_consensus.cc:2454] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a47e5c56f13843018ac584914a8d4707 in term 0.
I20240502 20:46:11.119988 6703 raft_consensus.cc:2454] T 681478951a8143f9959b78eec15efaf1 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a47e5c56f13843018ac584914a8d4707 in term 0.
I20240502 20:46:11.120057 7099 raft_consensus.cc:1261] T 15a2cbe8285a4938be7f628f7f8b734d P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:11.120277 6902 leader_election.cc:302] T 681478951a8143f9959b78eec15efaf1 P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7845c397230d49358ba2fad9db53f97c, a47e5c56f13843018ac584914a8d4707; no voters:
I20240502 20:46:11.120617 9156 raft_consensus.cc:2781] T 681478951a8143f9959b78eec15efaf1 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:11.120685 9156 raft_consensus.cc:482] T 681478951a8143f9959b78eec15efaf1 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:11.120723 9156 raft_consensus.cc:3037] T 681478951a8143f9959b78eec15efaf1 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:11.121341 8918 consensus_queue.cc:1035] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:11.121538 4638 raft_consensus.cc:1261] T 15a2cbe8285a4938be7f628f7f8b734d P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:11.121821 9156 raft_consensus.cc:504] T 681478951a8143f9959b78eec15efaf1 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:11.121899 8918 consensus_queue.cc:1035] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:11.122195 9156 leader_election.cc:288] T 681478951a8143f9959b78eec15efaf1 P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 1 election: Requested vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:46:11.122697 4377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681478951a8143f9959b78eec15efaf1" candidate_uuid: "a47e5c56f13843018ac584914a8d4707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2"
I20240502 20:46:11.122707 6703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681478951a8143f9959b78eec15efaf1" candidate_uuid: "a47e5c56f13843018ac584914a8d4707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7845c397230d49358ba2fad9db53f97c"
I20240502 20:46:11.122818 4377 raft_consensus.cc:3037] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:11.122819 6703 raft_consensus.cc:3037] T 681478951a8143f9959b78eec15efaf1 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:11.123764 4377 raft_consensus.cc:2454] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a47e5c56f13843018ac584914a8d4707 in term 1.
I20240502 20:46:11.123826 6703 raft_consensus.cc:2454] T 681478951a8143f9959b78eec15efaf1 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a47e5c56f13843018ac584914a8d4707 in term 1.
I20240502 20:46:11.124085 6902 leader_election.cc:302] T 681478951a8143f9959b78eec15efaf1 P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7845c397230d49358ba2fad9db53f97c, a47e5c56f13843018ac584914a8d4707; no voters:
I20240502 20:46:11.124311 9156 raft_consensus.cc:2781] T 681478951a8143f9959b78eec15efaf1 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:11.124394 9156 raft_consensus.cc:686] T 681478951a8143f9959b78eec15efaf1 P a47e5c56f13843018ac584914a8d4707 [term 1 LEADER]: Becoming Leader. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Running, Role: LEADER
I20240502 20:46:11.124516 9156 consensus_queue.cc:237] T 681478951a8143f9959b78eec15efaf1 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:11.124558 7937 raft_consensus.cc:2932] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:11.125674 7099 raft_consensus.cc:2932] T 15a2cbe8285a4938be7f628f7f8b734d P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:11.126798 4222 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 15a2cbe8285a4938be7f628f7f8b734d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:11.126854 4638 raft_consensus.cc:2932] T 15a2cbe8285a4938be7f628f7f8b734d P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:11.127837 4227 catalog_manager.cc:5484] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2 reported cstate change: config changed from index 3 to 4, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New cstate: current_term: 1 leader_uuid: "c92e6e46782849479c8a0c6d82276fb2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:11.127851 4231 catalog_manager.cc:5484] T 681478951a8143f9959b78eec15efaf1 P a47e5c56f13843018ac584914a8d4707 reported cstate change: term changed from 0 to 1, leader changed from <none> to a47e5c56f13843018ac584914a8d4707 (127.30.32.71). New cstate: current_term: 1 leader_uuid: "a47e5c56f13843018ac584914a8d4707" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: HEALTHY } } }
I20240502 20:46:11.131714 4487 tablet_service.cc:1514] Processing DeleteTablet for tablet 15a2cbe8285a4938be7f628f7f8b734d with delete_type TABLET_DATA_TOMBSTONED (TS c3d26eafd51547acb780d4aac5ebe7cc not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48890
I20240502 20:46:11.132462 9609 tablet_replica.cc:330] T 15a2cbe8285a4938be7f628f7f8b734d P c3d26eafd51547acb780d4aac5ebe7cc: stopping tablet replica
I20240502 20:46:11.132642 9609 raft_consensus.cc:2229] T 15a2cbe8285a4938be7f628f7f8b734d P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:11.132822 9609 raft_consensus.cc:2258] T 15a2cbe8285a4938be7f628f7f8b734d P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:11.133435 9609 ts_tablet_manager.cc:1875] T 15a2cbe8285a4938be7f628f7f8b734d P c3d26eafd51547acb780d4aac5ebe7cc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:11.135823 9609 ts_tablet_manager.cc:1888] T 15a2cbe8285a4938be7f628f7f8b734d P c3d26eafd51547acb780d4aac5ebe7cc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 20:46:11.135903 9609 log.cc:1196] T 15a2cbe8285a4938be7f628f7f8b734d P c3d26eafd51547acb780d4aac5ebe7cc: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal/wals/15a2cbe8285a4938be7f628f7f8b734d
I20240502 20:46:11.136518 4221 catalog_manager.cc:4830] TS c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097): tablet 15a2cbe8285a4938be7f628f7f8b734d (table rebalancer_extra_table_2 [id=44d455c601714afb8333ce2027e68a4a]) successfully deleted
I20240502 20:46:11.138180 9610 ts_tablet_manager.cc:902] T 28da7347f982468281570ffa7503a854 P a4bf1df7fa6b4948be502c182bfbb833: Initiating tablet copy from peer 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:46:11.138623 9610 tablet_copy_client.cc:308] T 28da7347f982468281570ffa7503a854 P a4bf1df7fa6b4948be502c182bfbb833: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.67:42141
I20240502 20:46:11.139078 4658 tablet_copy_service.cc:140] P 923cfab926e7443f8ffd7b21ad0a9d91: Received BeginTabletCopySession request for tablet 28da7347f982468281570ffa7503a854 from peer a4bf1df7fa6b4948be502c182bfbb833 ({username='slave'} at 127.30.32.70:32901)
I20240502 20:46:11.139180 4658 tablet_copy_service.cc:161] P 923cfab926e7443f8ffd7b21ad0a9d91: Beginning new tablet copy session on tablet 28da7347f982468281570ffa7503a854 from peer a4bf1df7fa6b4948be502c182bfbb833 at {username='slave'} at 127.30.32.70:32901: session id = a4bf1df7fa6b4948be502c182bfbb833-28da7347f982468281570ffa7503a854
I20240502 20:46:11.139896 4658 tablet_copy_source_session.cc:205] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:11.140645 9610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28da7347f982468281570ffa7503a854. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:11.142510 9610 tablet_copy_client.cc:783] T 28da7347f982468281570ffa7503a854 P a4bf1df7fa6b4948be502c182bfbb833: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:11.142652 9610 tablet_copy_client.cc:647] T 28da7347f982468281570ffa7503a854 P a4bf1df7fa6b4948be502c182bfbb833: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:11.143813 9610 tablet_copy_client.cc:523] T 28da7347f982468281570ffa7503a854 P a4bf1df7fa6b4948be502c182bfbb833: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:11.145198 9610 tablet_bootstrap.cc:492] T 28da7347f982468281570ffa7503a854 P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap starting.
I20240502 20:46:11.163324 9610 tablet_bootstrap.cc:492] T 28da7347f982468281570ffa7503a854 P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:11.163636 9610 tablet_bootstrap.cc:492] T 28da7347f982468281570ffa7503a854 P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap complete.
I20240502 20:46:11.163813 9610 ts_tablet_manager.cc:1367] T 28da7347f982468281570ffa7503a854 P a4bf1df7fa6b4948be502c182bfbb833: Time spent bootstrapping tablet: real 0.019s user 0.016s sys 0.000s
I20240502 20:46:11.164223 9610 raft_consensus.cc:348] T 28da7347f982468281570ffa7503a854 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } }
I20240502 20:46:11.164352 9610 raft_consensus.cc:729] T 28da7347f982468281570ffa7503a854 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Initialized, Role: LEARNER
I20240502 20:46:11.164459 9610 consensus_queue.cc:260] T 28da7347f982468281570ffa7503a854 P a4bf1df7fa6b4948be502c182bfbb833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } }
I20240502 20:46:11.164737 9610 ts_tablet_manager.cc:1398] T 28da7347f982468281570ffa7503a854 P a4bf1df7fa6b4948be502c182bfbb833: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:11.165256 4658 tablet_copy_service.cc:342] P 923cfab926e7443f8ffd7b21ad0a9d91: Request end of tablet copy session a4bf1df7fa6b4948be502c182bfbb833-28da7347f982468281570ffa7503a854 received from {username='slave'} at 127.30.32.70:32901
I20240502 20:46:11.165354 4658 tablet_copy_service.cc:433] P 923cfab926e7443f8ffd7b21ad0a9d91: ending tablet copy session a4bf1df7fa6b4948be502c182bfbb833-28da7347f982468281570ffa7503a854 on tablet 28da7347f982468281570ffa7503a854 with peer a4bf1df7fa6b4948be502c182bfbb833
I20240502 20:46:11.168350 9616 ts_tablet_manager.cc:902] T d6c2626b3ee7493c8a349f08e17082b5 P c3d26eafd51547acb780d4aac5ebe7cc: Initiating tablet copy from peer 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:46:11.169059 9616 tablet_copy_client.cc:308] T d6c2626b3ee7493c8a349f08e17082b5 P c3d26eafd51547acb780d4aac5ebe7cc: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.67:42141
I20240502 20:46:11.169549 4658 tablet_copy_service.cc:140] P 923cfab926e7443f8ffd7b21ad0a9d91: Received BeginTabletCopySession request for tablet d6c2626b3ee7493c8a349f08e17082b5 from peer c3d26eafd51547acb780d4aac5ebe7cc ({username='slave'} at 127.30.32.66:46555)
I20240502 20:46:11.169656 4658 tablet_copy_service.cc:161] P 923cfab926e7443f8ffd7b21ad0a9d91: Beginning new tablet copy session on tablet d6c2626b3ee7493c8a349f08e17082b5 from peer c3d26eafd51547acb780d4aac5ebe7cc at {username='slave'} at 127.30.32.66:46555: session id = c3d26eafd51547acb780d4aac5ebe7cc-d6c2626b3ee7493c8a349f08e17082b5
I20240502 20:46:11.170428 4658 tablet_copy_source_session.cc:205] T d6c2626b3ee7493c8a349f08e17082b5 P 923cfab926e7443f8ffd7b21ad0a9d91: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:11.171221 9616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6c2626b3ee7493c8a349f08e17082b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:11.173250 9616 tablet_copy_client.cc:783] T d6c2626b3ee7493c8a349f08e17082b5 P c3d26eafd51547acb780d4aac5ebe7cc: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:11.173408 9616 tablet_copy_client.cc:647] T d6c2626b3ee7493c8a349f08e17082b5 P c3d26eafd51547acb780d4aac5ebe7cc: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:11.174585 9616 tablet_copy_client.cc:523] T d6c2626b3ee7493c8a349f08e17082b5 P c3d26eafd51547acb780d4aac5ebe7cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:11.175195 7017 raft_consensus.cc:1050] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91: attempting to promote NON_VOTER 7845c397230d49358ba2fad9db53f97c to VOTER
I20240502 20:46:11.175601 7017 consensus_queue.cc:237] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } }
I20240502 20:46:11.175948 9616 tablet_bootstrap.cc:492] T d6c2626b3ee7493c8a349f08e17082b5 P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:46:11.177147 6967 raft_consensus.cc:1261] T 8565ebbd21dd4f708a0254d1ed2d6670 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:11.177533 4507 raft_consensus.cc:1261] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:11.177948 8236 consensus_queue.cc:1035] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:11.178299 8236 consensus_queue.cc:1035] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:11.178963 6703 raft_consensus.cc:1261] T 8565ebbd21dd4f708a0254d1ed2d6670 P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:11.179327 7017 consensus_queue.cc:1035] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:11.179651 9616 tablet_bootstrap.cc:492] T d6c2626b3ee7493c8a349f08e17082b5 P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:11.179919 9616 tablet_bootstrap.cc:492] T d6c2626b3ee7493c8a349f08e17082b5 P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap complete.
I20240502 20:46:11.180078 9616 ts_tablet_manager.cc:1367] T d6c2626b3ee7493c8a349f08e17082b5 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.004s user 0.005s sys 0.000s
I20240502 20:46:11.180598 9616 raft_consensus.cc:348] T d6c2626b3ee7493c8a349f08e17082b5 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } }
I20240502 20:46:11.180696 9616 raft_consensus.cc:729] T d6c2626b3ee7493c8a349f08e17082b5 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: LEARNER
I20240502 20:46:11.180790 9616 consensus_queue.cc:260] T d6c2626b3ee7493c8a349f08e17082b5 P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } }
I20240502 20:46:11.181143 9616 ts_tablet_manager.cc:1398] T d6c2626b3ee7493c8a349f08e17082b5 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:11.181389 8168 raft_consensus.cc:2932] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:11.181762 6703 raft_consensus.cc:2932] T 8565ebbd21dd4f708a0254d1ed2d6670 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:11.183792 6967 raft_consensus.cc:2932] T 8565ebbd21dd4f708a0254d1ed2d6670 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:11.184530 4228 catalog_manager.cc:5484] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index 7 to 8, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:11.184795 4507 raft_consensus.cc:2932] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:11.186234 7967 raft_consensus.cc:1050] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc: attempting to promote NON_VOTER 201d2b6485084e7dad2cbb722af94779 to VOTER
I20240502 20:46:11.186604 7967 consensus_queue.cc:237] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } }
I20240502 20:46:11.187951 4768 raft_consensus.cc:1261] T d6fb394fdcff436886fe0634d79c024c P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:11.188484 7099 raft_consensus.cc:1261] T d6fb394fdcff436886fe0634d79c024c P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:11.188517 6703 raft_consensus.cc:1261] T d6fb394fdcff436886fe0634d79c024c P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:11.188841 7967 consensus_queue.cc:1035] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:11.189870 6375 consensus_queue.cc:1035] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:11.190243 6375 consensus_queue.cc:1035] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:11.192150 6375 raft_consensus.cc:2932] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:11.192600 4768 raft_consensus.cc:2932] T d6fb394fdcff436886fe0634d79c024c P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:11.193933 6703 raft_consensus.cc:2932] T d6fb394fdcff436886fe0634d79c024c P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:11.193926 7099 raft_consensus.cc:2932] T d6fb394fdcff436886fe0634d79c024c P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:11.194222 4658 tablet_copy_service.cc:342] P 923cfab926e7443f8ffd7b21ad0a9d91: Request end of tablet copy session c3d26eafd51547acb780d4aac5ebe7cc-d6c2626b3ee7493c8a349f08e17082b5 received from {username='slave'} at 127.30.32.66:46555
I20240502 20:46:11.194311 4658 tablet_copy_service.cc:433] P 923cfab926e7443f8ffd7b21ad0a9d91: ending tablet copy session c3d26eafd51547acb780d4aac5ebe7cc-d6c2626b3ee7493c8a349f08e17082b5 on tablet d6c2626b3ee7493c8a349f08e17082b5 with peer c3d26eafd51547acb780d4aac5ebe7cc
I20240502 20:46:11.195560 4230 catalog_manager.cc:5484] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: config changed from index 7 to 8, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:11.198354 4507 consensus_queue.cc:237] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } }
I20240502 20:46:11.198652 4228 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41812:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 20:46:11.199414 6703 raft_consensus.cc:1261] T d6fb394fdcff436886fe0634d79c024c P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:46:11.199779 8907 consensus_queue.cc:1035] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:11.200034 7099 raft_consensus.cc:1261] T d6fb394fdcff436886fe0634d79c024c P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:46:11.200368 8907 consensus_queue.cc:1035] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:11.201721 8907 raft_consensus.cc:2932] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:11.203773 4221 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d6fb394fdcff436886fe0634d79c024c with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:11.203938 6815 tablet_service.cc:1467] Processing CreateTablet for tablet a609406c51704d8688e0d6ccfe344ec8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7e280a862b96486b888d929cf955cc94]), partition=
I20240502 20:46:11.204216 6815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a609406c51704d8688e0d6ccfe344ec8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:11.204226 6681 tablet_service.cc:1467] Processing CreateTablet for tablet a609406c51704d8688e0d6ccfe344ec8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7e280a862b96486b888d929cf955cc94]), partition=
I20240502 20:46:11.204150 7099 raft_consensus.cc:2932] T d6fb394fdcff436886fe0634d79c024c P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:11.204473 6681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a609406c51704d8688e0d6ccfe344ec8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:11.206156 9379 tablet_bootstrap.cc:492] T a609406c51704d8688e0d6ccfe344ec8 P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap starting.
I20240502 20:46:11.206404 6947 tablet_service.cc:1467] Processing CreateTablet for tablet a609406c51704d8688e0d6ccfe344ec8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7e280a862b96486b888d929cf955cc94]), partition=
I20240502 20:46:11.206504 9263 tablet_bootstrap.cc:492] T a609406c51704d8688e0d6ccfe344ec8 P 7845c397230d49358ba2fad9db53f97c: Bootstrap starting.
I20240502 20:46:11.206640 6947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a609406c51704d8688e0d6ccfe344ec8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:11.207093 9379 tablet_bootstrap.cc:654] T a609406c51704d8688e0d6ccfe344ec8 P a4bf1df7fa6b4948be502c182bfbb833: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:11.207329 9263 tablet_bootstrap.cc:654] T a609406c51704d8688e0d6ccfe344ec8 P 7845c397230d49358ba2fad9db53f97c: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:11.208585 9379 tablet_bootstrap.cc:492] T a609406c51704d8688e0d6ccfe344ec8 P a4bf1df7fa6b4948be502c182bfbb833: No bootstrap required, opened a new log
I20240502 20:46:11.208671 9379 ts_tablet_manager.cc:1367] T a609406c51704d8688e0d6ccfe344ec8 P a4bf1df7fa6b4948be502c182bfbb833: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:46:11.209028 9610 ts_tablet_manager.cc:902] T a62a7ce526f14c25994bb9e0e3c3bc84 P a4bf1df7fa6b4948be502c182bfbb833: Initiating tablet copy from peer c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097)
I20240502 20:46:11.209093 9263 tablet_bootstrap.cc:492] T a609406c51704d8688e0d6ccfe344ec8 P 7845c397230d49358ba2fad9db53f97c: No bootstrap required, opened a new log
I20240502 20:46:11.209174 9263 ts_tablet_manager.cc:1367] T a609406c51704d8688e0d6ccfe344ec8 P 7845c397230d49358ba2fad9db53f97c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:46:11.209342 9379 raft_consensus.cc:348] T a609406c51704d8688e0d6ccfe344ec8 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:11.209436 9379 raft_consensus.cc:374] T a609406c51704d8688e0d6ccfe344ec8 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:11.209448 9610 tablet_copy_client.cc:308] T a62a7ce526f14c25994bb9e0e3c3bc84 P a4bf1df7fa6b4948be502c182bfbb833: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.66:42097
I20240502 20:46:11.209470 9379 raft_consensus.cc:729] T a609406c51704d8688e0d6ccfe344ec8 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Initialized, Role: FOLLOWER
I20240502 20:46:11.209580 9379 consensus_queue.cc:260] T a609406c51704d8688e0d6ccfe344ec8 P a4bf1df7fa6b4948be502c182bfbb833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:11.209609 9263 raft_consensus.cc:348] T a609406c51704d8688e0d6ccfe344ec8 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:11.209692 9263 raft_consensus.cc:374] T a609406c51704d8688e0d6ccfe344ec8 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:11.209728 9263 raft_consensus.cc:729] T a609406c51704d8688e0d6ccfe344ec8 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Initialized, Role: FOLLOWER
I20240502 20:46:11.209856 9263 consensus_queue.cc:260] T a609406c51704d8688e0d6ccfe344ec8 P 7845c397230d49358ba2fad9db53f97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:11.210107 9263 ts_tablet_manager.cc:1398] T a609406c51704d8688e0d6ccfe344ec8 P 7845c397230d49358ba2fad9db53f97c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:11.210114 9379 ts_tablet_manager.cc:1398] T a609406c51704d8688e0d6ccfe344ec8 P a4bf1df7fa6b4948be502c182bfbb833: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:11.213210 9385 tablet_bootstrap.cc:492] T a609406c51704d8688e0d6ccfe344ec8 P a47e5c56f13843018ac584914a8d4707: Bootstrap starting.
I20240502 20:46:11.214150 9385 tablet_bootstrap.cc:654] T a609406c51704d8688e0d6ccfe344ec8 P a47e5c56f13843018ac584914a8d4707: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:11.215305 9385 tablet_bootstrap.cc:492] T a609406c51704d8688e0d6ccfe344ec8 P a47e5c56f13843018ac584914a8d4707: No bootstrap required, opened a new log
I20240502 20:46:11.215400 9385 ts_tablet_manager.cc:1367] T a609406c51704d8688e0d6ccfe344ec8 P a47e5c56f13843018ac584914a8d4707: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:11.215886 9385 raft_consensus.cc:348] T a609406c51704d8688e0d6ccfe344ec8 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:11.215996 9385 raft_consensus.cc:374] T a609406c51704d8688e0d6ccfe344ec8 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:11.216037 9385 raft_consensus.cc:729] T a609406c51704d8688e0d6ccfe344ec8 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Initialized, Role: FOLLOWER
I20240502 20:46:11.216161 9385 consensus_queue.cc:260] T a609406c51704d8688e0d6ccfe344ec8 P a47e5c56f13843018ac584914a8d4707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:11.216111 4227 catalog_manager.cc:5484] T d6fb394fdcff436886fe0634d79c024c P 201d2b6485084e7dad2cbb722af94779 reported cstate change: config changed from index 8 to 9, VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) evicted. New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:11.217130 6703 raft_consensus.cc:2932] T d6fb394fdcff436886fe0634d79c024c P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:11.216954 4528 tablet_copy_service.cc:140] P c3d26eafd51547acb780d4aac5ebe7cc: Received BeginTabletCopySession request for tablet a62a7ce526f14c25994bb9e0e3c3bc84 from peer a4bf1df7fa6b4948be502c182bfbb833 ({username='slave'} at 127.30.32.70:46909)
I20240502 20:46:11.218546 4528 tablet_copy_service.cc:161] P c3d26eafd51547acb780d4aac5ebe7cc: Beginning new tablet copy session on tablet a62a7ce526f14c25994bb9e0e3c3bc84 from peer a4bf1df7fa6b4948be502c182bfbb833 at {username='slave'} at 127.30.32.70:46909: session id = a4bf1df7fa6b4948be502c182bfbb833-a62a7ce526f14c25994bb9e0e3c3bc84
I20240502 20:46:11.219516 4528 tablet_copy_source_session.cc:205] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:11.219867 4750 tablet_service.cc:1514] Processing DeleteTablet for tablet d6fb394fdcff436886fe0634d79c024c with delete_type TABLET_DATA_TOMBSTONED (TS ed47c023452a497192f35d2f617bf9bd not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:38526
I20240502 20:46:11.220230 7299 tablet_replica.cc:330] T d6fb394fdcff436886fe0634d79c024c P ed47c023452a497192f35d2f617bf9bd: stopping tablet replica
I20240502 20:46:11.220394 7299 raft_consensus.cc:2229] T d6fb394fdcff436886fe0634d79c024c P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:11.220615 7299 raft_consensus.cc:2258] T d6fb394fdcff436886fe0634d79c024c P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:11.220746 9385 ts_tablet_manager.cc:1398] T a609406c51704d8688e0d6ccfe344ec8 P a47e5c56f13843018ac584914a8d4707: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 20:46:11.221338 9610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a62a7ce526f14c25994bb9e0e3c3bc84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:11.223644 9610 tablet_copy_client.cc:783] T a62a7ce526f14c25994bb9e0e3c3bc84 P a4bf1df7fa6b4948be502c182bfbb833: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:11.223794 9610 tablet_copy_client.cc:647] T a62a7ce526f14c25994bb9e0e3c3bc84 P a4bf1df7fa6b4948be502c182bfbb833: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:11.223982 9535 ts_tablet_manager.cc:902] T 20373b53281a4214b27f014396f7a651 P 201d2b6485084e7dad2cbb722af94779: Initiating tablet copy from peer ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:46:11.224519 9535 tablet_copy_client.cc:308] T 20373b53281a4214b27f014396f7a651 P 201d2b6485084e7dad2cbb722af94779: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.68:38193
I20240502 20:46:11.225090 4790 tablet_copy_service.cc:140] P ed47c023452a497192f35d2f617bf9bd: Received BeginTabletCopySession request for tablet 20373b53281a4214b27f014396f7a651 from peer 201d2b6485084e7dad2cbb722af94779 ({username='slave'} at 127.30.32.72:39947)
I20240502 20:46:11.225194 4790 tablet_copy_service.cc:161] P ed47c023452a497192f35d2f617bf9bd: Beginning new tablet copy session on tablet 20373b53281a4214b27f014396f7a651 from peer 201d2b6485084e7dad2cbb722af94779 at {username='slave'} at 127.30.32.72:39947: session id = 201d2b6485084e7dad2cbb722af94779-20373b53281a4214b27f014396f7a651
I20240502 20:46:11.226475 4790 tablet_copy_source_session.cc:205] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:11.226662 9033 raft_consensus.cc:482] T a609406c51704d8688e0d6ccfe344ec8 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:11.226748 9033 raft_consensus.cc:504] T a609406c51704d8688e0d6ccfe344ec8 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:11.227105 9033 leader_election.cc:288] T a609406c51704d8688e0d6ccfe344ec8 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
I20240502 20:46:11.227525 6967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a609406c51704d8688e0d6ccfe344ec8" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47e5c56f13843018ac584914a8d4707" is_pre_election: true
I20240502 20:46:11.227602 6703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a609406c51704d8688e0d6ccfe344ec8" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7845c397230d49358ba2fad9db53f97c" is_pre_election: true
I20240502 20:46:11.227665 6967 raft_consensus.cc:2454] T a609406c51704d8688e0d6ccfe344ec8 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 0.
I20240502 20:46:11.227758 6703 raft_consensus.cc:2454] T a609406c51704d8688e0d6ccfe344ec8 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 0.
I20240502 20:46:11.228060 6770 leader_election.cc:302] T a609406c51704d8688e0d6ccfe344ec8 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7845c397230d49358ba2fad9db53f97c, a4bf1df7fa6b4948be502c182bfbb833; no voters:
I20240502 20:46:11.228224 9535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20373b53281a4214b27f014396f7a651. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:11.228246 9033 raft_consensus.cc:2781] T a609406c51704d8688e0d6ccfe344ec8 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:11.228297 9033 raft_consensus.cc:482] T a609406c51704d8688e0d6ccfe344ec8 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:11.228329 9033 raft_consensus.cc:3037] T a609406c51704d8688e0d6ccfe344ec8 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:11.229316 9033 raft_consensus.cc:504] T a609406c51704d8688e0d6ccfe344ec8 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:11.229669 9033 leader_election.cc:288] T a609406c51704d8688e0d6ccfe344ec8 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 election: Requested vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
I20240502 20:46:11.230090 6703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a609406c51704d8688e0d6ccfe344ec8" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7845c397230d49358ba2fad9db53f97c"
I20240502 20:46:11.230203 6703 raft_consensus.cc:3037] T a609406c51704d8688e0d6ccfe344ec8 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:11.230322 6967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a609406c51704d8688e0d6ccfe344ec8" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47e5c56f13843018ac584914a8d4707"
I20240502 20:46:11.230435 6967 raft_consensus.cc:3037] T a609406c51704d8688e0d6ccfe344ec8 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:11.231221 6703 raft_consensus.cc:2454] T a609406c51704d8688e0d6ccfe344ec8 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 1.
I20240502 20:46:11.231503 6967 raft_consensus.cc:2454] T a609406c51704d8688e0d6ccfe344ec8 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 1.
I20240502 20:46:11.231897 6771 leader_election.cc:302] T a609406c51704d8688e0d6ccfe344ec8 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a47e5c56f13843018ac584914a8d4707, a4bf1df7fa6b4948be502c182bfbb833; no voters:
I20240502 20:46:11.232053 9033 raft_consensus.cc:2781] T a609406c51704d8688e0d6ccfe344ec8 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:11.232142 9033 raft_consensus.cc:686] T a609406c51704d8688e0d6ccfe344ec8 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEADER]: Becoming Leader. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Running, Role: LEADER
I20240502 20:46:11.232278 9033 consensus_queue.cc:237] T a609406c51704d8688e0d6ccfe344ec8 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:11.234535 4231 catalog_manager.cc:5484] T a609406c51704d8688e0d6ccfe344ec8 P a4bf1df7fa6b4948be502c182bfbb833 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70). New cstate: current_term: 1 leader_uuid: "a4bf1df7fa6b4948be502c182bfbb833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:11.237012 7299 ts_tablet_manager.cc:1875] T d6fb394fdcff436886fe0634d79c024c P ed47c023452a497192f35d2f617bf9bd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:11.237967 9535 tablet_copy_client.cc:783] T 20373b53281a4214b27f014396f7a651 P 201d2b6485084e7dad2cbb722af94779: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:11.238126 9535 tablet_copy_client.cc:647] T 20373b53281a4214b27f014396f7a651 P 201d2b6485084e7dad2cbb722af94779: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:11.238180 9610 tablet_copy_client.cc:523] T a62a7ce526f14c25994bb9e0e3c3bc84 P a4bf1df7fa6b4948be502c182bfbb833: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:11.239948 9610 tablet_bootstrap.cc:492] T a62a7ce526f14c25994bb9e0e3c3bc84 P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap starting.
I20240502 20:46:11.240692 7299 ts_tablet_manager.cc:1888] T d6fb394fdcff436886fe0634d79c024c P ed47c023452a497192f35d2f617bf9bd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 20:46:11.240798 9535 tablet_copy_client.cc:523] T 20373b53281a4214b27f014396f7a651 P 201d2b6485084e7dad2cbb722af94779: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:11.240907 7299 log.cc:1196] T d6fb394fdcff436886fe0634d79c024c P ed47c023452a497192f35d2f617bf9bd: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal/wals/d6fb394fdcff436886fe0634d79c024c
I20240502 20:46:11.241513 4222 catalog_manager.cc:4830] TS ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193): tablet d6fb394fdcff436886fe0634d79c024c (table rebalance_test_table_9 [id=7c90eeb05db84afb8f78ab7f14e0ced4]) successfully deleted
I20240502 20:46:11.242230 9535 tablet_bootstrap.cc:492] T 20373b53281a4214b27f014396f7a651 P 201d2b6485084e7dad2cbb722af94779: Bootstrap starting.
I20240502 20:46:11.249208 9642 ts_tablet_manager.cc:902] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd: Initiating tablet copy from peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
I20240502 20:46:11.258558 9514 ts_tablet_manager.cc:902] T 93b7f7de1f8c49389c3d8c1e03c67f26 P 7845c397230d49358ba2fad9db53f97c: Initiating tablet copy from peer c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:46:11.259140 9514 tablet_copy_client.cc:308] T 93b7f7de1f8c49389c3d8c1e03c67f26 P 7845c397230d49358ba2fad9db53f97c: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.65:33827
I20240502 20:46:11.259778 4397 tablet_copy_service.cc:140] P c92e6e46782849479c8a0c6d82276fb2: Received BeginTabletCopySession request for tablet 93b7f7de1f8c49389c3d8c1e03c67f26 from peer 7845c397230d49358ba2fad9db53f97c ({username='slave'} at 127.30.32.69:38945)
I20240502 20:46:11.259905 4397 tablet_copy_service.cc:161] P c92e6e46782849479c8a0c6d82276fb2: Beginning new tablet copy session on tablet 93b7f7de1f8c49389c3d8c1e03c67f26 from peer 7845c397230d49358ba2fad9db53f97c at {username='slave'} at 127.30.32.69:38945: session id = 7845c397230d49358ba2fad9db53f97c-93b7f7de1f8c49389c3d8c1e03c67f26
I20240502 20:46:11.259889 9642 tablet_copy_client.cc:308] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.71:46841
I20240502 20:46:11.260541 6987 tablet_copy_service.cc:140] P a47e5c56f13843018ac584914a8d4707: Received BeginTabletCopySession request for tablet ef4002126e8847809add182f262d62d9 from peer ed47c023452a497192f35d2f617bf9bd ({username='slave'} at 127.30.32.68:35313)
I20240502 20:46:11.260650 6987 tablet_copy_service.cc:161] P a47e5c56f13843018ac584914a8d4707: Beginning new tablet copy session on tablet ef4002126e8847809add182f262d62d9 from peer ed47c023452a497192f35d2f617bf9bd at {username='slave'} at 127.30.32.68:35313: session id = ed47c023452a497192f35d2f617bf9bd-ef4002126e8847809add182f262d62d9
I20240502 20:46:11.260921 4397 tablet_copy_source_session.cc:205] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:11.261723 6987 tablet_copy_source_session.cc:205] T ef4002126e8847809add182f262d62d9 P a47e5c56f13843018ac584914a8d4707: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:11.262102 9514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93b7f7de1f8c49389c3d8c1e03c67f26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:11.264379 9514 tablet_copy_client.cc:783] T 93b7f7de1f8c49389c3d8c1e03c67f26 P 7845c397230d49358ba2fad9db53f97c: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:11.264457 9642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef4002126e8847809add182f262d62d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:11.264531 9514 tablet_copy_client.cc:647] T 93b7f7de1f8c49389c3d8c1e03c67f26 P 7845c397230d49358ba2fad9db53f97c: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:11.264874 7967 consensus_queue.cc:1035] T 6c8f977981c54fa1bf5d173e83608146 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:11.268574 9642 tablet_copy_client.cc:783] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:11.268770 9642 tablet_copy_client.cc:647] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:11.270316 9642 tablet_copy_client.cc:523] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:11.271939 9642 tablet_bootstrap.cc:492] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:46:11.272461 9514 tablet_copy_client.cc:523] T 93b7f7de1f8c49389c3d8c1e03c67f26 P 7845c397230d49358ba2fad9db53f97c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:11.273805 9514 tablet_bootstrap.cc:492] T 93b7f7de1f8c49389c3d8c1e03c67f26 P 7845c397230d49358ba2fad9db53f97c: Bootstrap starting.
I20240502 20:46:11.275468 9610 tablet_bootstrap.cc:492] T a62a7ce526f14c25994bb9e0e3c3bc84 P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:11.275753 9610 tablet_bootstrap.cc:492] T a62a7ce526f14c25994bb9e0e3c3bc84 P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap complete.
I20240502 20:46:11.275929 9610 ts_tablet_manager.cc:1367] T a62a7ce526f14c25994bb9e0e3c3bc84 P a4bf1df7fa6b4948be502c182bfbb833: Time spent bootstrapping tablet: real 0.036s user 0.030s sys 0.000s
I20240502 20:46:11.276315 9610 raft_consensus.cc:348] T a62a7ce526f14c25994bb9e0e3c3bc84 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } }
I20240502 20:46:11.276425 9610 raft_consensus.cc:729] T a62a7ce526f14c25994bb9e0e3c3bc84 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Initialized, Role: LEARNER
I20240502 20:46:11.276522 9610 consensus_queue.cc:260] T a62a7ce526f14c25994bb9e0e3c3bc84 P a4bf1df7fa6b4948be502c182bfbb833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } }
I20240502 20:46:11.276832 9610 ts_tablet_manager.cc:1398] T a62a7ce526f14c25994bb9e0e3c3bc84 P a4bf1df7fa6b4948be502c182bfbb833: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:11.277326 4528 tablet_copy_service.cc:342] P c3d26eafd51547acb780d4aac5ebe7cc: Request end of tablet copy session a4bf1df7fa6b4948be502c182bfbb833-a62a7ce526f14c25994bb9e0e3c3bc84 received from {username='slave'} at 127.30.32.70:46909
I20240502 20:46:11.277407 4528 tablet_copy_service.cc:433] P c3d26eafd51547acb780d4aac5ebe7cc: ending tablet copy session a4bf1df7fa6b4948be502c182bfbb833-a62a7ce526f14c25994bb9e0e3c3bc84 on tablet a62a7ce526f14c25994bb9e0e3c3bc84 with peer a4bf1df7fa6b4948be502c182bfbb833
I20240502 20:46:11.279152 8146 consensus_queue.cc:1035] T 6c8f977981c54fa1bf5d173e83608146 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:11.283622 8460 consensus_queue.cc:1035] T 51232fccd98c461fb3d9016081f52ba5 P 201d2b6485084e7dad2cbb722af94779 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:11.285892 9514 tablet_bootstrap.cc:492] T 93b7f7de1f8c49389c3d8c1e03c67f26 P 7845c397230d49358ba2fad9db53f97c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:11.286175 9514 tablet_bootstrap.cc:492] T 93b7f7de1f8c49389c3d8c1e03c67f26 P 7845c397230d49358ba2fad9db53f97c: Bootstrap complete.
I20240502 20:46:11.286372 9514 ts_tablet_manager.cc:1367] T 93b7f7de1f8c49389c3d8c1e03c67f26 P 7845c397230d49358ba2fad9db53f97c: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240502 20:46:11.286782 9514 raft_consensus.cc:348] T 93b7f7de1f8c49389c3d8c1e03c67f26 P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } }
I20240502 20:46:11.286882 9514 raft_consensus.cc:729] T 93b7f7de1f8c49389c3d8c1e03c67f26 P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Initialized, Role: LEARNER
I20240502 20:46:11.286968 9514 consensus_queue.cc:260] T 93b7f7de1f8c49389c3d8c1e03c67f26 P 7845c397230d49358ba2fad9db53f97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } }
I20240502 20:46:11.287266 9514 ts_tablet_manager.cc:1398] T 93b7f7de1f8c49389c3d8c1e03c67f26 P 7845c397230d49358ba2fad9db53f97c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:11.287706 4397 tablet_copy_service.cc:342] P c92e6e46782849479c8a0c6d82276fb2: Request end of tablet copy session 7845c397230d49358ba2fad9db53f97c-93b7f7de1f8c49389c3d8c1e03c67f26 received from {username='slave'} at 127.30.32.69:38945
I20240502 20:46:11.287783 4397 tablet_copy_service.cc:433] P c92e6e46782849479c8a0c6d82276fb2: ending tablet copy session 7845c397230d49358ba2fad9db53f97c-93b7f7de1f8c49389c3d8c1e03c67f26 on tablet 93b7f7de1f8c49389c3d8c1e03c67f26 with peer 7845c397230d49358ba2fad9db53f97c
I20240502 20:46:11.288586 9642 tablet_bootstrap.cc:492] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:11.288858 9642 tablet_bootstrap.cc:492] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd: Bootstrap complete.
I20240502 20:46:11.289024 9642 ts_tablet_manager.cc:1367] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.004s
I20240502 20:46:11.289408 9642 raft_consensus.cc:348] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: true } }
I20240502 20:46:11.289513 9642 raft_consensus.cc:729] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: LEARNER
I20240502 20:46:11.289604 9642 consensus_queue.cc:260] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: true } }
I20240502 20:46:11.290254 9413 consensus_queue.cc:1035] T 51232fccd98c461fb3d9016081f52ba5 P 201d2b6485084e7dad2cbb722af94779 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:11.290729 9642 ts_tablet_manager.cc:1398] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:11.291242 6987 tablet_copy_service.cc:342] P a47e5c56f13843018ac584914a8d4707: Request end of tablet copy session ed47c023452a497192f35d2f617bf9bd-ef4002126e8847809add182f262d62d9 received from {username='slave'} at 127.30.32.68:35313
I20240502 20:46:11.291322 6987 tablet_copy_service.cc:433] P a47e5c56f13843018ac584914a8d4707: ending tablet copy session ed47c023452a497192f35d2f617bf9bd-ef4002126e8847809add182f262d62d9 on tablet ef4002126e8847809add182f262d62d9 with peer ed47c023452a497192f35d2f617bf9bd
I20240502 20:46:11.295785 9535 tablet_bootstrap.cc:492] T 20373b53281a4214b27f014396f7a651 P 201d2b6485084e7dad2cbb722af94779: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:11.296207 9535 tablet_bootstrap.cc:492] T 20373b53281a4214b27f014396f7a651 P 201d2b6485084e7dad2cbb722af94779: Bootstrap complete.
I20240502 20:46:11.296427 9535 ts_tablet_manager.cc:1367] T 20373b53281a4214b27f014396f7a651 P 201d2b6485084e7dad2cbb722af94779: Time spent bootstrapping tablet: real 0.054s user 0.029s sys 0.000s
I20240502 20:46:11.296897 9535 raft_consensus.cc:348] T 20373b53281a4214b27f014396f7a651 P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } }
I20240502 20:46:11.297055 9535 raft_consensus.cc:729] T 20373b53281a4214b27f014396f7a651 P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Initialized, Role: LEARNER
I20240502 20:46:11.297160 9535 consensus_queue.cc:260] T 20373b53281a4214b27f014396f7a651 P 201d2b6485084e7dad2cbb722af94779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } }
I20240502 20:46:11.297436 9535 ts_tablet_manager.cc:1398] T 20373b53281a4214b27f014396f7a651 P 201d2b6485084e7dad2cbb722af94779: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:11.298087 4790 tablet_copy_service.cc:342] P ed47c023452a497192f35d2f617bf9bd: Request end of tablet copy session 201d2b6485084e7dad2cbb722af94779-20373b53281a4214b27f014396f7a651 received from {username='slave'} at 127.30.32.72:39947
I20240502 20:46:11.298170 4790 tablet_copy_service.cc:433] P ed47c023452a497192f35d2f617bf9bd: ending tablet copy session 201d2b6485084e7dad2cbb722af94779-20373b53281a4214b27f014396f7a651 on tablet 20373b53281a4214b27f014396f7a651 with peer 201d2b6485084e7dad2cbb722af94779
I20240502 20:46:11.313102 9663 ts_tablet_manager.cc:902] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91: Initiating tablet copy from peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
I20240502 20:46:11.313620 9663 tablet_copy_client.cc:308] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.71:46841
I20240502 20:46:11.314136 6987 tablet_copy_service.cc:140] P a47e5c56f13843018ac584914a8d4707: Received BeginTabletCopySession request for tablet 8602022a45e1482ebf8dab9930ce8209 from peer 923cfab926e7443f8ffd7b21ad0a9d91 ({username='slave'} at 127.30.32.67:59801)
I20240502 20:46:11.314249 6987 tablet_copy_service.cc:161] P a47e5c56f13843018ac584914a8d4707: Beginning new tablet copy session on tablet 8602022a45e1482ebf8dab9930ce8209 from peer 923cfab926e7443f8ffd7b21ad0a9d91 at {username='slave'} at 127.30.32.67:59801: session id = 923cfab926e7443f8ffd7b21ad0a9d91-8602022a45e1482ebf8dab9930ce8209
I20240502 20:46:11.315145 6987 tablet_copy_source_session.cc:205] T 8602022a45e1482ebf8dab9930ce8209 P a47e5c56f13843018ac584914a8d4707: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:11.316035 9663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8602022a45e1482ebf8dab9930ce8209. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:11.318176 9663 tablet_copy_client.cc:783] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:11.318332 9663 tablet_copy_client.cc:647] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:11.319403 9663 tablet_copy_client.cc:523] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:11.320367 9663 tablet_bootstrap.cc:492] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:46:11.323113 9663 tablet_bootstrap.cc:492] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:11.323354 9663 tablet_bootstrap.cc:492] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap complete.
I20240502 20:46:11.323504 9663 ts_tablet_manager.cc:1367] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:46:11.323875 9663 raft_consensus.cc:348] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } }
I20240502 20:46:11.323976 9663 raft_consensus.cc:729] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: LEARNER
I20240502 20:46:11.324072 9663 consensus_queue.cc:260] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } }
I20240502 20:46:11.324337 9663 ts_tablet_manager.cc:1398] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:11.325946 6987 tablet_copy_service.cc:342] P a47e5c56f13843018ac584914a8d4707: Request end of tablet copy session 923cfab926e7443f8ffd7b21ad0a9d91-8602022a45e1482ebf8dab9930ce8209 received from {username='slave'} at 127.30.32.67:59801
I20240502 20:46:11.326035 6987 tablet_copy_service.cc:433] P a47e5c56f13843018ac584914a8d4707: ending tablet copy session 923cfab926e7443f8ffd7b21ad0a9d91-8602022a45e1482ebf8dab9930ce8209 on tablet 8602022a45e1482ebf8dab9930ce8209 with peer 923cfab926e7443f8ffd7b21ad0a9d91
I20240502 20:46:11.339668 9535 ts_tablet_manager.cc:902] T a7ea68e8581a4cd5a6212abc96dfbcde P 201d2b6485084e7dad2cbb722af94779: Initiating tablet copy from peer 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:46:11.340170 9535 tablet_copy_client.cc:308] T a7ea68e8581a4cd5a6212abc96dfbcde P 201d2b6485084e7dad2cbb722af94779: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.67:42141
I20240502 20:46:11.340662 4658 tablet_copy_service.cc:140] P 923cfab926e7443f8ffd7b21ad0a9d91: Received BeginTabletCopySession request for tablet a7ea68e8581a4cd5a6212abc96dfbcde from peer 201d2b6485084e7dad2cbb722af94779 ({username='slave'} at 127.30.32.72:36609)
I20240502 20:46:11.340795 4658 tablet_copy_service.cc:161] P 923cfab926e7443f8ffd7b21ad0a9d91: Beginning new tablet copy session on tablet a7ea68e8581a4cd5a6212abc96dfbcde from peer 201d2b6485084e7dad2cbb722af94779 at {username='slave'} at 127.30.32.72:36609: session id = 201d2b6485084e7dad2cbb722af94779-a7ea68e8581a4cd5a6212abc96dfbcde
I20240502 20:46:11.341725 4658 tablet_copy_source_session.cc:205] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:11.342625 9535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7ea68e8581a4cd5a6212abc96dfbcde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:11.344218 4231 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 20:46:11.344766 9535 tablet_copy_client.cc:783] T a7ea68e8581a4cd5a6212abc96dfbcde P 201d2b6485084e7dad2cbb722af94779: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:11.344909 9535 tablet_copy_client.cc:647] T a7ea68e8581a4cd5a6212abc96dfbcde P 201d2b6485084e7dad2cbb722af94779: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:11.345952 9535 tablet_copy_client.cc:523] T a7ea68e8581a4cd5a6212abc96dfbcde P 201d2b6485084e7dad2cbb722af94779: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:11.347190 9535 tablet_bootstrap.cc:492] T a7ea68e8581a4cd5a6212abc96dfbcde P 201d2b6485084e7dad2cbb722af94779: Bootstrap starting.
I20240502 20:46:11.350152 6703 raft_consensus.cc:1261] T a609406c51704d8688e0d6ccfe344ec8 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Refusing update from remote peer a4bf1df7fa6b4948be502c182bfbb833: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:11.350196 6967 raft_consensus.cc:1261] T a609406c51704d8688e0d6ccfe344ec8 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Refusing update from remote peer a4bf1df7fa6b4948be502c182bfbb833: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:11.350585 9402 consensus_queue.cc:1035] T a609406c51704d8688e0d6ccfe344ec8 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:11.350850 9033 consensus_queue.cc:1035] T a609406c51704d8688e0d6ccfe344ec8 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:11.353940 4228 catalog_manager.cc:5576] TS 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429) does not have the latest schema for tablet a609406c51704d8688e0d6ccfe344ec8 (table rebalancer_dynamic_table [id=7e280a862b96486b888d929cf955cc94]). Expected version 1 got 0
I20240502 20:46:11.354462 9364 tablet.cc:1722] T a609406c51704d8688e0d6ccfe344ec8 P 7845c397230d49358ba2fad9db53f97c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 20:46:11.354769 9364 tablet.cc:1620] T a609406c51704d8688e0d6ccfe344ec8 P 7845c397230d49358ba2fad9db53f97c: MemRowSet was empty: no flush needed.
I20240502 20:46:11.356436 4227 catalog_manager.cc:5576] TS a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867) does not have the latest schema for tablet a609406c51704d8688e0d6ccfe344ec8 (table rebalancer_dynamic_table [id=7e280a862b96486b888d929cf955cc94]). Expected version 1 got 0
I20240502 20:46:11.359055 4228 catalog_manager.cc:5576] TS a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841) does not have the latest schema for tablet a609406c51704d8688e0d6ccfe344ec8 (table rebalancer_dynamic_table [id=7e280a862b96486b888d929cf955cc94]). Expected version 1 got 0
I20240502 20:46:11.359571 9687 tablet.cc:1722] T a609406c51704d8688e0d6ccfe344ec8 P a4bf1df7fa6b4948be502c182bfbb833: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 20:46:11.359860 9687 tablet.cc:1620] T a609406c51704d8688e0d6ccfe344ec8 P a4bf1df7fa6b4948be502c182bfbb833: MemRowSet was empty: no flush needed.
I20240502 20:46:11.360255 4227 catalog_manager.cc:5576] TS a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867) does not have the latest schema for tablet a609406c51704d8688e0d6ccfe344ec8 (table rebalancer_dynamic_table [id=7e280a862b96486b888d929cf955cc94]). Expected version 1 got 0
I20240502 20:46:11.361785 9692 tablet.cc:1722] T a609406c51704d8688e0d6ccfe344ec8 P a47e5c56f13843018ac584914a8d4707: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 20:46:11.362143 4228 catalog_manager.cc:5576] TS a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841) does not have the latest schema for tablet a609406c51704d8688e0d6ccfe344ec8 (table rebalancer_dynamic_table [id=7e280a862b96486b888d929cf955cc94]). Expected version 1 got 0
I20240502 20:46:11.362587 9692 tablet.cc:1620] T a609406c51704d8688e0d6ccfe344ec8 P a47e5c56f13843018ac584914a8d4707: MemRowSet was empty: no flush needed.
I20240502 20:46:11.362830 4231 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: rebalancer_dynamic_table [id=7e280a862b96486b888d929cf955cc94] alter complete (version 1)
I20240502 20:46:11.374315 9535 tablet_bootstrap.cc:492] T a7ea68e8581a4cd5a6212abc96dfbcde P 201d2b6485084e7dad2cbb722af94779: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:11.374581 9535 tablet_bootstrap.cc:492] T a7ea68e8581a4cd5a6212abc96dfbcde P 201d2b6485084e7dad2cbb722af94779: Bootstrap complete.
I20240502 20:46:11.374778 9535 ts_tablet_manager.cc:1367] T a7ea68e8581a4cd5a6212abc96dfbcde P 201d2b6485084e7dad2cbb722af94779: Time spent bootstrapping tablet: real 0.028s user 0.021s sys 0.000s
I20240502 20:46:11.375402 9535 raft_consensus.cc:348] T a7ea68e8581a4cd5a6212abc96dfbcde P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } }
I20240502 20:46:11.375553 9535 raft_consensus.cc:729] T a7ea68e8581a4cd5a6212abc96dfbcde P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Initialized, Role: LEARNER
I20240502 20:46:11.375679 9535 consensus_queue.cc:260] T a7ea68e8581a4cd5a6212abc96dfbcde P 201d2b6485084e7dad2cbb722af94779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } }
I20240502 20:46:11.375946 9535 ts_tablet_manager.cc:1398] T a7ea68e8581a4cd5a6212abc96dfbcde P 201d2b6485084e7dad2cbb722af94779: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:11.376731 4658 tablet_copy_service.cc:342] P 923cfab926e7443f8ffd7b21ad0a9d91: Request end of tablet copy session 201d2b6485084e7dad2cbb722af94779-a7ea68e8581a4cd5a6212abc96dfbcde received from {username='slave'} at 127.30.32.72:36609
I20240502 20:46:11.376816 4658 tablet_copy_service.cc:433] P 923cfab926e7443f8ffd7b21ad0a9d91: ending tablet copy session 201d2b6485084e7dad2cbb722af94779-a7ea68e8581a4cd5a6212abc96dfbcde on tablet a7ea68e8581a4cd5a6212abc96dfbcde with peer 201d2b6485084e7dad2cbb722af94779
I20240502 20:46:11.435468 4638 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8565ebbd21dd4f708a0254d1ed2d6670"
dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33434
I20240502 20:46:11.435634 4638 raft_consensus.cc:595] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Received request to transfer leadership
I20240502 20:46:11.444623 9616 ts_tablet_manager.cc:902] T 52e302882311488ab1cac90ad03fe270 P c3d26eafd51547acb780d4aac5ebe7cc: Initiating tablet copy from peer a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867)
I20240502 20:46:11.445144 9616 tablet_copy_client.cc:308] T 52e302882311488ab1cac90ad03fe270 P c3d26eafd51547acb780d4aac5ebe7cc: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.70:41867
I20240502 20:46:11.445683 6848 tablet_copy_service.cc:140] P a4bf1df7fa6b4948be502c182bfbb833: Received BeginTabletCopySession request for tablet 52e302882311488ab1cac90ad03fe270 from peer c3d26eafd51547acb780d4aac5ebe7cc ({username='slave'} at 127.30.32.66:55279)
I20240502 20:46:11.445785 6848 tablet_copy_service.cc:161] P a4bf1df7fa6b4948be502c182bfbb833: Beginning new tablet copy session on tablet 52e302882311488ab1cac90ad03fe270 from peer c3d26eafd51547acb780d4aac5ebe7cc at {username='slave'} at 127.30.32.66:55279: session id = c3d26eafd51547acb780d4aac5ebe7cc-52e302882311488ab1cac90ad03fe270
I20240502 20:46:11.446642 6848 tablet_copy_source_session.cc:205] T 52e302882311488ab1cac90ad03fe270 P a4bf1df7fa6b4948be502c182bfbb833: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:11.447602 9616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52e302882311488ab1cac90ad03fe270. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:11.450050 9616 tablet_copy_client.cc:783] T 52e302882311488ab1cac90ad03fe270 P c3d26eafd51547acb780d4aac5ebe7cc: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:11.450209 9616 tablet_copy_client.cc:647] T 52e302882311488ab1cac90ad03fe270 P c3d26eafd51547acb780d4aac5ebe7cc: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:11.451771 9616 tablet_copy_client.cc:523] T 52e302882311488ab1cac90ad03fe270 P c3d26eafd51547acb780d4aac5ebe7cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:11.453143 9616 tablet_bootstrap.cc:492] T 52e302882311488ab1cac90ad03fe270 P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:46:11.457077 9616 tablet_bootstrap.cc:492] T 52e302882311488ab1cac90ad03fe270 P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:11.457329 9616 tablet_bootstrap.cc:492] T 52e302882311488ab1cac90ad03fe270 P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap complete.
I20240502 20:46:11.457587 9616 ts_tablet_manager.cc:1367] T 52e302882311488ab1cac90ad03fe270 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 20:46:11.458447 9616 raft_consensus.cc:348] T 52e302882311488ab1cac90ad03fe270 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } }
I20240502 20:46:11.458606 9616 raft_consensus.cc:729] T 52e302882311488ab1cac90ad03fe270 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: LEARNER
I20240502 20:46:11.458742 9616 consensus_queue.cc:260] T 52e302882311488ab1cac90ad03fe270 P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } }
I20240502 20:46:11.459071 9616 ts_tablet_manager.cc:1398] T 52e302882311488ab1cac90ad03fe270 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:11.460904 6848 tablet_copy_service.cc:342] P a4bf1df7fa6b4948be502c182bfbb833: Request end of tablet copy session c3d26eafd51547acb780d4aac5ebe7cc-52e302882311488ab1cac90ad03fe270 received from {username='slave'} at 127.30.32.66:55279
I20240502 20:46:11.461007 6848 tablet_copy_service.cc:433] P a4bf1df7fa6b4948be502c182bfbb833: ending tablet copy session c3d26eafd51547acb780d4aac5ebe7cc-52e302882311488ab1cac90ad03fe270 on tablet 52e302882311488ab1cac90ad03fe270 with peer c3d26eafd51547acb780d4aac5ebe7cc
I20240502 20:46:11.467120 4228 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 20:46:11.472517 9687 tablet.cc:1722] T a609406c51704d8688e0d6ccfe344ec8 P a4bf1df7fa6b4948be502c182bfbb833: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 20:46:11.472865 9687 tablet.cc:1620] T a609406c51704d8688e0d6ccfe344ec8 P a4bf1df7fa6b4948be502c182bfbb833: MemRowSet was empty: no flush needed.
I20240502 20:46:11.473408 9364 tablet.cc:1722] T a609406c51704d8688e0d6ccfe344ec8 P 7845c397230d49358ba2fad9db53f97c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 20:46:11.473686 9364 tablet.cc:1620] T a609406c51704d8688e0d6ccfe344ec8 P 7845c397230d49358ba2fad9db53f97c: MemRowSet was empty: no flush needed.
I20240502 20:46:11.474682 9692 tablet.cc:1722] T a609406c51704d8688e0d6ccfe344ec8 P a47e5c56f13843018ac584914a8d4707: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 20:46:11.474987 9692 tablet.cc:1620] T a609406c51704d8688e0d6ccfe344ec8 P a47e5c56f13843018ac584914a8d4707: MemRowSet was empty: no flush needed.
I20240502 20:46:11.478612 6967 consensus_queue.cc:237] T bcfd626ea0a04fa7b03dd0ede6b35266 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: true } }
I20240502 20:46:11.478711 4227 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: rebalancer_dynamic_table [id=7e280a862b96486b888d929cf955cc94] alter complete (version 2)
I20240502 20:46:11.480257 7098 raft_consensus.cc:1261] T bcfd626ea0a04fa7b03dd0ede6b35266 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Refusing update from remote peer a47e5c56f13843018ac584914a8d4707: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:11.480407 6703 raft_consensus.cc:1261] T bcfd626ea0a04fa7b03dd0ede6b35266 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Refusing update from remote peer a47e5c56f13843018ac584914a8d4707: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 20:46:11.480406 6903 consensus_peers.cc:485] T bcfd626ea0a04fa7b03dd0ede6b35266 P a47e5c56f13843018ac584914a8d4707 -> Peer ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193): Couldn't send request to peer ed47c023452a497192f35d2f617bf9bd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bcfd626ea0a04fa7b03dd0ede6b35266. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:11.480803 9775 consensus_queue.cc:1035] T bcfd626ea0a04fa7b03dd0ede6b35266 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:46:11.480993 9155 consensus_queue.cc:1035] T bcfd626ea0a04fa7b03dd0ede6b35266 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:46:11.482774 9155 raft_consensus.cc:2932] T bcfd626ea0a04fa7b03dd0ede6b35266 P a47e5c56f13843018ac584914a8d4707 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: true } } }
I20240502 20:46:11.483212 6703 raft_consensus.cc:2932] T bcfd626ea0a04fa7b03dd0ede6b35266 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: true } } }
I20240502 20:46:11.483275 7098 raft_consensus.cc:2932] T bcfd626ea0a04fa7b03dd0ede6b35266 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: true } } }
I20240502 20:46:11.485213 4227 catalog_manager.cc:5484] T bcfd626ea0a04fa7b03dd0ede6b35266 P a47e5c56f13843018ac584914a8d4707 reported cstate change: config changed from index -1 to 2, NON_VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) added. New cstate: current_term: 1 leader_uuid: "a47e5c56f13843018ac584914a8d4707" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:11.492759 7098 raft_consensus.cc:1203] T 41f84d5db2034af591b56e109be67934 P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:11.495399 4377 consensus_queue.cc:237] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } }
I20240502 20:46:11.496917 4638 raft_consensus.cc:1261] T c9d25657e9a14b35b2f6e8d82f4533c7 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:11.496970 4768 raft_consensus.cc:1261] T c9d25657e9a14b35b2f6e8d82f4533c7 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 20:46:11.497073 4312 consensus_peers.cc:485] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c9d25657e9a14b35b2f6e8d82f4533c7. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:11.497349 6138 consensus_queue.cc:1035] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:11.497532 8918 consensus_queue.cc:1035] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:11.499491 8918 raft_consensus.cc:2932] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } } }
I20240502 20:46:11.499897 4768 raft_consensus.cc:2932] T c9d25657e9a14b35b2f6e8d82f4533c7 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } } }
I20240502 20:46:11.500020 4638 raft_consensus.cc:2932] T c9d25657e9a14b35b2f6e8d82f4533c7 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } } }
I20240502 20:46:11.502951 4231 catalog_manager.cc:5484] T c9d25657e9a14b35b2f6e8d82f4533c7 P ed47c023452a497192f35d2f617bf9bd reported cstate change: config changed from index -1 to 4, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New cstate: current_term: 1 leader_uuid: "c92e6e46782849479c8a0c6d82276fb2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } } }
I20240502 20:46:11.510615 4638 consensus_queue.cc:237] T e7e0539e60064d97a7eb0eacd9df065e P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } }
I20240502 20:46:11.511077 6703 raft_consensus.cc:1203] T 02302fc79dc04b0396041c600c2f4bef P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:11.512781 4507 raft_consensus.cc:1261] T e7e0539e60064d97a7eb0eacd9df065e P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 20:46:11.513141 4573 consensus_peers.cc:485] T e7e0539e60064d97a7eb0eacd9df065e P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867): Couldn't send request to peer a4bf1df7fa6b4948be502c182bfbb833. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e7e0539e60064d97a7eb0eacd9df065e. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:11.513166 4768 raft_consensus.cc:1261] T e7e0539e60064d97a7eb0eacd9df065e P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:11.513552 8236 consensus_queue.cc:1035] T e7e0539e60064d97a7eb0eacd9df065e P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:11.513713 9043 consensus_queue.cc:1035] T e7e0539e60064d97a7eb0eacd9df065e P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:11.515487 9041 raft_consensus.cc:2932] T e7e0539e60064d97a7eb0eacd9df065e P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } } }
I20240502 20:46:11.515959 4507 raft_consensus.cc:2932] T e7e0539e60064d97a7eb0eacd9df065e P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } } }
I20240502 20:46:11.516011 4768 raft_consensus.cc:2932] T e7e0539e60064d97a7eb0eacd9df065e P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } } }
I20240502 20:46:11.518615 4228 catalog_manager.cc:5484] T e7e0539e60064d97a7eb0eacd9df065e P ed47c023452a497192f35d2f617bf9bd reported cstate change: config changed from index -1 to 4, NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) added. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } } }
I20240502 20:46:11.549132 6835 raft_consensus.cc:1203] T 28da7347f982468281570ffa7503a854 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:11.551486 9556 ts_tablet_manager.cc:902] T d37fab152a8a44868d3160ec3057bb98 P a47e5c56f13843018ac584914a8d4707: Initiating tablet copy from peer ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:46:11.551993 9556 tablet_copy_client.cc:308] T d37fab152a8a44868d3160ec3057bb98 P a47e5c56f13843018ac584914a8d4707: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.68:38193
I20240502 20:46:11.552466 4790 tablet_copy_service.cc:140] P ed47c023452a497192f35d2f617bf9bd: Received BeginTabletCopySession request for tablet d37fab152a8a44868d3160ec3057bb98 from peer a47e5c56f13843018ac584914a8d4707 ({username='slave'} at 127.30.32.71:49469)
I20240502 20:46:11.552597 4790 tablet_copy_service.cc:161] P ed47c023452a497192f35d2f617bf9bd: Beginning new tablet copy session on tablet d37fab152a8a44868d3160ec3057bb98 from peer a47e5c56f13843018ac584914a8d4707 at {username='slave'} at 127.30.32.71:49469: session id = a47e5c56f13843018ac584914a8d4707-d37fab152a8a44868d3160ec3057bb98
I20240502 20:46:11.553561 4790 tablet_copy_source_session.cc:205] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:11.554533 9556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d37fab152a8a44868d3160ec3057bb98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:11.556895 9556 tablet_copy_client.cc:783] T d37fab152a8a44868d3160ec3057bb98 P a47e5c56f13843018ac584914a8d4707: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:11.557509 9556 tablet_copy_client.cc:647] T d37fab152a8a44868d3160ec3057bb98 P a47e5c56f13843018ac584914a8d4707: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:11.558926 9556 tablet_copy_client.cc:523] T d37fab152a8a44868d3160ec3057bb98 P a47e5c56f13843018ac584914a8d4707: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:11.560143 9556 tablet_bootstrap.cc:492] T d37fab152a8a44868d3160ec3057bb98 P a47e5c56f13843018ac584914a8d4707: Bootstrap starting.
I20240502 20:46:11.567518 9042 raft_consensus.cc:979] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91: : Instructing follower c3d26eafd51547acb780d4aac5ebe7cc to start an election
I20240502 20:46:11.567620 9042 raft_consensus.cc:1067] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Signalling peer c3d26eafd51547acb780d4aac5ebe7cc to start an election
I20240502 20:46:11.567981 6967 raft_consensus.cc:1203] T 00eb91e624554baf918e96b098019be8 P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 20:46:11.568220 4507 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8565ebbd21dd4f708a0254d1ed2d6670"
dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc"
from {username='slave'} at 127.30.32.67:59037
I20240502 20:46:11.568379 4507 raft_consensus.cc:482] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:46:11.568444 4507 raft_consensus.cc:3037] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:11.569365 4507 raft_consensus.cc:504] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } }
I20240502 20:46:11.570585 4638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8565ebbd21dd4f708a0254d1ed2d6670" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91"
I20240502 20:46:11.570711 4638 raft_consensus.cc:3032] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:46:11.570760 4638 raft_consensus.cc:729] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Running, Role: LEADER
I20240502 20:46:11.570931 4638 consensus_queue.cc:260] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } }
I20240502 20:46:11.571123 4638 raft_consensus.cc:3037] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:11.571825 4638 raft_consensus.cc:2454] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 2.
I20240502 20:46:11.572242 4507 leader_election.cc:288] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 2 election: Requested vote from peers 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
I20240502 20:46:11.572672 6703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8565ebbd21dd4f708a0254d1ed2d6670" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "7845c397230d49358ba2fad9db53f97c"
I20240502 20:46:11.572824 6703 raft_consensus.cc:3037] T 8565ebbd21dd4f708a0254d1ed2d6670 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:11.573837 6703 raft_consensus.cc:2454] T 8565ebbd21dd4f708a0254d1ed2d6670 P 7845c397230d49358ba2fad9db53f97c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 2.
I20240502 20:46:11.574158 4443 leader_election.cc:302] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 7845c397230d49358ba2fad9db53f97c, 923cfab926e7443f8ffd7b21ad0a9d91, c3d26eafd51547acb780d4aac5ebe7cc; no voters:
I20240502 20:46:11.574330 8146 raft_consensus.cc:2781] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:46:11.574425 8146 raft_consensus.cc:686] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc [term 2 LEADER]: Becoming Leader. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Running, Role: LEADER
I20240502 20:46:11.574445 6967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8565ebbd21dd4f708a0254d1ed2d6670" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "a47e5c56f13843018ac584914a8d4707"
I20240502 20:46:11.574549 6967 raft_consensus.cc:3037] T 8565ebbd21dd4f708a0254d1ed2d6670 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:11.574576 8146 consensus_queue.cc:237] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } }
I20240502 20:46:11.575439 6967 raft_consensus.cc:2454] T 8565ebbd21dd4f708a0254d1ed2d6670 P a47e5c56f13843018ac584914a8d4707 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 2.
I20240502 20:46:11.576378 4227 catalog_manager.cc:5484] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: term changed from 1 to 2, leader changed from 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) to c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66). New cstate: current_term: 2 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:11.582041 4227 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 20:46:11.587393 9687 tablet.cc:1722] T a609406c51704d8688e0d6ccfe344ec8 P a4bf1df7fa6b4948be502c182bfbb833: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 20:46:11.587968 9692 tablet.cc:1722] T a609406c51704d8688e0d6ccfe344ec8 P a47e5c56f13843018ac584914a8d4707: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 20:46:11.589473 9556 tablet_bootstrap.cc:492] T d37fab152a8a44868d3160ec3057bb98 P a47e5c56f13843018ac584914a8d4707: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:11.589797 9556 tablet_bootstrap.cc:492] T d37fab152a8a44868d3160ec3057bb98 P a47e5c56f13843018ac584914a8d4707: Bootstrap complete.
I20240502 20:46:11.589951 9556 ts_tablet_manager.cc:1367] T d37fab152a8a44868d3160ec3057bb98 P a47e5c56f13843018ac584914a8d4707: Time spent bootstrapping tablet: real 0.030s user 0.024s sys 0.004s
I20240502 20:46:11.590683 9364 tablet.cc:1722] T a609406c51704d8688e0d6ccfe344ec8 P 7845c397230d49358ba2fad9db53f97c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 20:46:11.591804 9556 raft_consensus.cc:348] T d37fab152a8a44868d3160ec3057bb98 P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } }
I20240502 20:46:11.591979 9556 raft_consensus.cc:729] T d37fab152a8a44868d3160ec3057bb98 P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Initialized, Role: LEARNER
I20240502 20:46:11.592111 9556 consensus_queue.cc:260] T d37fab152a8a44868d3160ec3057bb98 P a47e5c56f13843018ac584914a8d4707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } }
I20240502 20:46:11.592475 9556 ts_tablet_manager.cc:1398] T d37fab152a8a44868d3160ec3057bb98 P a47e5c56f13843018ac584914a8d4707: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:11.593206 4790 tablet_copy_service.cc:342] P ed47c023452a497192f35d2f617bf9bd: Request end of tablet copy session a47e5c56f13843018ac584914a8d4707-d37fab152a8a44868d3160ec3057bb98 received from {username='slave'} at 127.30.32.71:49469
I20240502 20:46:11.593291 4790 tablet_copy_service.cc:433] P ed47c023452a497192f35d2f617bf9bd: ending tablet copy session a47e5c56f13843018ac584914a8d4707-d37fab152a8a44868d3160ec3057bb98 on tablet d37fab152a8a44868d3160ec3057bb98 with peer a47e5c56f13843018ac584914a8d4707
I20240502 20:46:11.593369 4231 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: rebalancer_dynamic_table_new_name [id=7e280a862b96486b888d929cf955cc94] alter complete (version 3)
I20240502 20:46:11.611114 4507 raft_consensus.cc:1203] T d6c2626b3ee7493c8a349f08e17082b5 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 20:46:11.614907 9156 consensus_queue.cc:1035] T 681478951a8143f9959b78eec15efaf1 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:11.618698 9156 consensus_queue.cc:1035] T 681478951a8143f9959b78eec15efaf1 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:11.660826 6835 raft_consensus.cc:1203] T a62a7ce526f14c25994bb9e0e3c3bc84 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 20:46:11.667763 4228 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41812:
name: "rebalancer_extra_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:46:11.674135 6681 tablet_service.cc:1467] Processing CreateTablet for tablet a135d04d8c3248bc954a07e5b6396fde (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=319cff57d33e47f69faa288db2be515b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:11.674486 6681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a135d04d8c3248bc954a07e5b6396fde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:11.674567 6683 tablet_service.cc:1467] Processing CreateTablet for tablet 9258667ccc3e4ef8b26fe75531dd4591 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=319cff57d33e47f69faa288db2be515b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:11.674878 6683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9258667ccc3e4ef8b26fe75531dd4591. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:11.675117 6815 tablet_service.cc:1467] Processing CreateTablet for tablet a999f994b78a4b33a95ee7d5407cba20 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=319cff57d33e47f69faa288db2be515b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:11.675364 6815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a999f994b78a4b33a95ee7d5407cba20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:11.676784 7079 tablet_service.cc:1467] Processing CreateTablet for tablet a135d04d8c3248bc954a07e5b6396fde (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=319cff57d33e47f69faa288db2be515b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:11.677054 7079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a135d04d8c3248bc954a07e5b6396fde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:11.679172 9887 tablet_bootstrap.cc:492] T a135d04d8c3248bc954a07e5b6396fde P 201d2b6485084e7dad2cbb722af94779: Bootstrap starting.
I20240502 20:46:11.679211 9263 tablet_bootstrap.cc:492] T a135d04d8c3248bc954a07e5b6396fde P 7845c397230d49358ba2fad9db53f97c: Bootstrap starting.
I20240502 20:46:11.679956 9887 tablet_bootstrap.cc:654] T a135d04d8c3248bc954a07e5b6396fde P 201d2b6485084e7dad2cbb722af94779: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:11.680028 9263 tablet_bootstrap.cc:654] T a135d04d8c3248bc954a07e5b6396fde P 7845c397230d49358ba2fad9db53f97c: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:11.681171 9379 tablet_bootstrap.cc:492] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap starting.
I20240502 20:46:11.681536 9263 tablet_bootstrap.cc:492] T a135d04d8c3248bc954a07e5b6396fde P 7845c397230d49358ba2fad9db53f97c: No bootstrap required, opened a new log
I20240502 20:46:11.681602 9263 ts_tablet_manager.cc:1367] T a135d04d8c3248bc954a07e5b6396fde P 7845c397230d49358ba2fad9db53f97c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:11.681949 9379 tablet_bootstrap.cc:654] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:11.681972 9263 raft_consensus.cc:348] T a135d04d8c3248bc954a07e5b6396fde P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:11.682073 9263 raft_consensus.cc:374] T a135d04d8c3248bc954a07e5b6396fde P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:11.682099 9263 raft_consensus.cc:729] T a135d04d8c3248bc954a07e5b6396fde P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Initialized, Role: FOLLOWER
I20240502 20:46:11.682190 9263 consensus_queue.cc:260] T a135d04d8c3248bc954a07e5b6396fde P 7845c397230d49358ba2fad9db53f97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:11.682444 9263 ts_tablet_manager.cc:1398] T a135d04d8c3248bc954a07e5b6396fde P 7845c397230d49358ba2fad9db53f97c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:11.683364 9263 tablet_bootstrap.cc:492] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c: Bootstrap starting.
I20240502 20:46:11.683328 6683 tablet_service.cc:1467] Processing CreateTablet for tablet a999f994b78a4b33a95ee7d5407cba20 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=319cff57d33e47f69faa288db2be515b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:11.683543 6683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a999f994b78a4b33a95ee7d5407cba20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:11.684486 9263 tablet_bootstrap.cc:654] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:11.684502 6946 tablet_service.cc:1467] Processing CreateTablet for tablet a999f994b78a4b33a95ee7d5407cba20 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=319cff57d33e47f69faa288db2be515b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:11.684785 6946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a999f994b78a4b33a95ee7d5407cba20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:11.686019 9379 tablet_bootstrap.cc:492] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833: No bootstrap required, opened a new log
I20240502 20:46:11.686120 9379 ts_tablet_manager.cc:1367] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 20:46:11.686342 6947 tablet_service.cc:1467] Processing CreateTablet for tablet 9258667ccc3e4ef8b26fe75531dd4591 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=319cff57d33e47f69faa288db2be515b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:11.686465 9263 tablet_bootstrap.cc:492] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c: No bootstrap required, opened a new log
I20240502 20:46:11.686547 6947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9258667ccc3e4ef8b26fe75531dd4591. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:11.686553 9263 ts_tablet_manager.cc:1367] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:46:11.686622 9379 raft_consensus.cc:348] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:11.686728 9379 raft_consensus.cc:374] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:11.686770 9379 raft_consensus.cc:729] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Initialized, Role: FOLLOWER
I20240502 20:46:11.686884 9379 consensus_queue.cc:260] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:11.687004 9263 raft_consensus.cc:348] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:11.687083 9263 raft_consensus.cc:374] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:11.687112 9263 raft_consensus.cc:729] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Initialized, Role: FOLLOWER
I20240502 20:46:11.687170 9379 ts_tablet_manager.cc:1398] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:11.687227 9263 consensus_queue.cc:260] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:11.687387 4356 tablet_service.cc:1467] Processing CreateTablet for tablet 9258667ccc3e4ef8b26fe75531dd4591 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=319cff57d33e47f69faa288db2be515b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:11.687470 9385 tablet_bootstrap.cc:492] T a999f994b78a4b33a95ee7d5407cba20 P a47e5c56f13843018ac584914a8d4707: Bootstrap starting.
I20240502 20:46:11.687512 9263 ts_tablet_manager.cc:1398] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:11.687639 9263 tablet_bootstrap.cc:492] T a999f994b78a4b33a95ee7d5407cba20 P 7845c397230d49358ba2fad9db53f97c: Bootstrap starting.
I20240502 20:46:11.687680 4356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9258667ccc3e4ef8b26fe75531dd4591. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:11.688422 9385 tablet_bootstrap.cc:654] T a999f994b78a4b33a95ee7d5407cba20 P a47e5c56f13843018ac584914a8d4707: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:11.688442 9263 tablet_bootstrap.cc:654] T a999f994b78a4b33a95ee7d5407cba20 P 7845c397230d49358ba2fad9db53f97c: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:11.688542 4354 tablet_service.cc:1467] Processing CreateTablet for tablet a135d04d8c3248bc954a07e5b6396fde (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=319cff57d33e47f69faa288db2be515b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:11.688804 4354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a135d04d8c3248bc954a07e5b6396fde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:11.691211 9887 tablet_bootstrap.cc:492] T a135d04d8c3248bc954a07e5b6396fde P 201d2b6485084e7dad2cbb722af94779: No bootstrap required, opened a new log
I20240502 20:46:11.691309 9887 ts_tablet_manager.cc:1367] T a135d04d8c3248bc954a07e5b6396fde P 201d2b6485084e7dad2cbb722af94779: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240502 20:46:11.691845 9887 raft_consensus.cc:348] T a135d04d8c3248bc954a07e5b6396fde P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:11.691957 9887 raft_consensus.cc:374] T a135d04d8c3248bc954a07e5b6396fde P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:11.691987 9887 raft_consensus.cc:729] T a135d04d8c3248bc954a07e5b6396fde P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Initialized, Role: FOLLOWER
I20240502 20:46:11.692116 9887 consensus_queue.cc:260] T a135d04d8c3248bc954a07e5b6396fde P 201d2b6485084e7dad2cbb722af94779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:11.692272 9263 tablet_bootstrap.cc:492] T a999f994b78a4b33a95ee7d5407cba20 P 7845c397230d49358ba2fad9db53f97c: No bootstrap required, opened a new log
I20240502 20:46:11.692342 9263 ts_tablet_manager.cc:1367] T a999f994b78a4b33a95ee7d5407cba20 P 7845c397230d49358ba2fad9db53f97c: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240502 20:46:11.692406 9887 ts_tablet_manager.cc:1398] T a135d04d8c3248bc954a07e5b6396fde P 201d2b6485084e7dad2cbb722af94779: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:11.692739 9263 raft_consensus.cc:348] T a999f994b78a4b33a95ee7d5407cba20 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:11.692823 9263 raft_consensus.cc:374] T a999f994b78a4b33a95ee7d5407cba20 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:11.692852 9263 raft_consensus.cc:729] T a999f994b78a4b33a95ee7d5407cba20 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Initialized, Role: FOLLOWER
I20240502 20:46:11.692955 9263 consensus_queue.cc:260] T a999f994b78a4b33a95ee7d5407cba20 P 7845c397230d49358ba2fad9db53f97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:11.693253 9263 ts_tablet_manager.cc:1398] T a999f994b78a4b33a95ee7d5407cba20 P 7845c397230d49358ba2fad9db53f97c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:11.693998 9385 tablet_bootstrap.cc:492] T a999f994b78a4b33a95ee7d5407cba20 P a47e5c56f13843018ac584914a8d4707: No bootstrap required, opened a new log
I20240502 20:46:11.694087 9385 ts_tablet_manager.cc:1367] T a999f994b78a4b33a95ee7d5407cba20 P a47e5c56f13843018ac584914a8d4707: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 20:46:11.694530 9385 raft_consensus.cc:348] T a999f994b78a4b33a95ee7d5407cba20 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:11.694628 9385 raft_consensus.cc:374] T a999f994b78a4b33a95ee7d5407cba20 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:11.694670 9385 raft_consensus.cc:729] T a999f994b78a4b33a95ee7d5407cba20 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Initialized, Role: FOLLOWER
I20240502 20:46:11.694803 9385 consensus_queue.cc:260] T a999f994b78a4b33a95ee7d5407cba20 P a47e5c56f13843018ac584914a8d4707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:11.695093 9385 ts_tablet_manager.cc:1398] T a999f994b78a4b33a95ee7d5407cba20 P a47e5c56f13843018ac584914a8d4707: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:11.695252 9385 tablet_bootstrap.cc:492] T 9258667ccc3e4ef8b26fe75531dd4591 P a47e5c56f13843018ac584914a8d4707: Bootstrap starting.
I20240502 20:46:11.695816 9894 tablet_bootstrap.cc:492] T 9258667ccc3e4ef8b26fe75531dd4591 P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:46:11.695962 4228 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 20:46:11.696048 4228 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 20:46:11.696102 9385 tablet_bootstrap.cc:654] T 9258667ccc3e4ef8b26fe75531dd4591 P a47e5c56f13843018ac584914a8d4707: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:11.697748 9894 tablet_bootstrap.cc:654] T 9258667ccc3e4ef8b26fe75531dd4591 P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:11.698078 4228 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: Sending DeleteTablet for 3 replicas of tablet a609406c51704d8688e0d6ccfe344ec8
I20240502 20:46:11.698350 9385 tablet_bootstrap.cc:492] T 9258667ccc3e4ef8b26fe75531dd4591 P a47e5c56f13843018ac584914a8d4707: No bootstrap required, opened a new log
I20240502 20:46:11.698423 9385 ts_tablet_manager.cc:1367] T 9258667ccc3e4ef8b26fe75531dd4591 P a47e5c56f13843018ac584914a8d4707: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:46:11.698719 6683 tablet_service.cc:1514] Processing DeleteTablet for tablet a609406c51704d8688e0d6ccfe344ec8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:11 UTC) from {username='slave'} at 127.0.0.1:37506
I20240502 20:46:11.698866 6815 tablet_service.cc:1514] Processing DeleteTablet for tablet a609406c51704d8688e0d6ccfe344ec8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:11 UTC) from {username='slave'} at 127.0.0.1:35290
I20240502 20:46:11.698853 9385 raft_consensus.cc:348] T 9258667ccc3e4ef8b26fe75531dd4591 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:11.698944 9385 raft_consensus.cc:374] T 9258667ccc3e4ef8b26fe75531dd4591 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:11.698972 9385 raft_consensus.cc:729] T 9258667ccc3e4ef8b26fe75531dd4591 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Initialized, Role: FOLLOWER
I20240502 20:46:11.699075 9385 consensus_queue.cc:260] T 9258667ccc3e4ef8b26fe75531dd4591 P a47e5c56f13843018ac584914a8d4707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:11.699097 6947 tablet_service.cc:1514] Processing DeleteTablet for tablet a609406c51704d8688e0d6ccfe344ec8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:11 UTC) from {username='slave'} at 127.0.0.1:45156
I20240502 20:46:11.699359 9898 tablet_replica.cc:330] T a609406c51704d8688e0d6ccfe344ec8 P a47e5c56f13843018ac584914a8d4707: stopping tablet replica
I20240502 20:46:11.699545 9898 raft_consensus.cc:2229] T a609406c51704d8688e0d6ccfe344ec8 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:11.699703 9898 raft_consensus.cc:2258] T a609406c51704d8688e0d6ccfe344ec8 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:11.699862 9105 raft_consensus.cc:482] T a135d04d8c3248bc954a07e5b6396fde P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:11.699949 9105 raft_consensus.cc:504] T a135d04d8c3248bc954a07e5b6396fde P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:11.700016 9385 ts_tablet_manager.cc:1398] T 9258667ccc3e4ef8b26fe75531dd4591 P a47e5c56f13843018ac584914a8d4707: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:11.700412 9105 leader_election.cc:288] T a135d04d8c3248bc954a07e5b6396fde P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
I20240502 20:46:11.700791 4377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a135d04d8c3248bc954a07e5b6396fde" candidate_uuid: "7845c397230d49358ba2fad9db53f97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2" is_pre_election: true
I20240502 20:46:11.700878 7098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a135d04d8c3248bc954a07e5b6396fde" candidate_uuid: "7845c397230d49358ba2fad9db53f97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201d2b6485084e7dad2cbb722af94779" is_pre_election: true
I20240502 20:46:11.701041 7098 raft_consensus.cc:2454] T a135d04d8c3248bc954a07e5b6396fde P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7845c397230d49358ba2fad9db53f97c in term 0.
W20240502 20:46:11.701290 6639 leader_election.cc:341] T a135d04d8c3248bc954a07e5b6396fde P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827): Illegal state: must be running to vote when last-logged opid is not known
I20240502 20:46:11.701388 9896 tablet_replica.cc:330] T a609406c51704d8688e0d6ccfe344ec8 P 7845c397230d49358ba2fad9db53f97c: stopping tablet replica
I20240502 20:46:11.701462 6636 leader_election.cc:302] T a135d04d8c3248bc954a07e5b6396fde P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 201d2b6485084e7dad2cbb722af94779, 7845c397230d49358ba2fad9db53f97c; no voters: c92e6e46782849479c8a0c6d82276fb2
I20240502 20:46:11.701519 9896 raft_consensus.cc:2229] T a609406c51704d8688e0d6ccfe344ec8 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:11.701624 9105 raft_consensus.cc:2781] T a135d04d8c3248bc954a07e5b6396fde P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:11.701694 9105 raft_consensus.cc:482] T a135d04d8c3248bc954a07e5b6396fde P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:11.701735 9105 raft_consensus.cc:3037] T a135d04d8c3248bc954a07e5b6396fde P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:11.701637 9896 raft_consensus.cc:2258] T a609406c51704d8688e0d6ccfe344ec8 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:11.702792 9105 raft_consensus.cc:504] T a135d04d8c3248bc954a07e5b6396fde P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:11.703153 9105 leader_election.cc:288] T a135d04d8c3248bc954a07e5b6396fde P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 election: Requested vote from peers c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
I20240502 20:46:11.703445 7098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a135d04d8c3248bc954a07e5b6396fde" candidate_uuid: "7845c397230d49358ba2fad9db53f97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201d2b6485084e7dad2cbb722af94779"
I20240502 20:46:11.703548 7098 raft_consensus.cc:3037] T a135d04d8c3248bc954a07e5b6396fde P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:11.703555 4377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a135d04d8c3248bc954a07e5b6396fde" candidate_uuid: "7845c397230d49358ba2fad9db53f97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2"
W20240502 20:46:11.703863 6639 leader_election.cc:341] T a135d04d8c3248bc954a07e5b6396fde P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827): Illegal state: must be running to vote when last-logged opid is not known
I20240502 20:46:11.704550 7098 raft_consensus.cc:2454] T a135d04d8c3248bc954a07e5b6396fde P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7845c397230d49358ba2fad9db53f97c in term 1.
I20240502 20:46:11.704692 9898 ts_tablet_manager.cc:1875] T a609406c51704d8688e0d6ccfe344ec8 P a47e5c56f13843018ac584914a8d4707: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:11.705174 6636 leader_election.cc:302] T a135d04d8c3248bc954a07e5b6396fde P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 201d2b6485084e7dad2cbb722af94779, 7845c397230d49358ba2fad9db53f97c; no voters: c92e6e46782849479c8a0c6d82276fb2
I20240502 20:46:11.705663 9105 raft_consensus.cc:2781] T a135d04d8c3248bc954a07e5b6396fde P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:11.705744 9105 raft_consensus.cc:686] T a135d04d8c3248bc954a07e5b6396fde P 7845c397230d49358ba2fad9db53f97c [term 1 LEADER]: Becoming Leader. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Running, Role: LEADER
I20240502 20:46:11.705828 9894 tablet_bootstrap.cc:492] T 9258667ccc3e4ef8b26fe75531dd4591 P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:46:11.705853 9105 consensus_queue.cc:237] T a135d04d8c3248bc954a07e5b6396fde P 7845c397230d49358ba2fad9db53f97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:11.705924 9894 ts_tablet_manager.cc:1367] T 9258667ccc3e4ef8b26fe75531dd4591 P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240502 20:46:11.706502 9894 raft_consensus.cc:348] T 9258667ccc3e4ef8b26fe75531dd4591 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:11.706624 9894 raft_consensus.cc:374] T 9258667ccc3e4ef8b26fe75531dd4591 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:11.706656 9894 raft_consensus.cc:729] T 9258667ccc3e4ef8b26fe75531dd4591 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:46:11.706780 9894 consensus_queue.cc:260] T 9258667ccc3e4ef8b26fe75531dd4591 P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:11.707082 9894 ts_tablet_manager.cc:1398] T 9258667ccc3e4ef8b26fe75531dd4591 P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:11.707239 9894 tablet_bootstrap.cc:492] T a135d04d8c3248bc954a07e5b6396fde P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:46:11.708145 9894 tablet_bootstrap.cc:654] T a135d04d8c3248bc954a07e5b6396fde P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:11.708307 4228 catalog_manager.cc:5484] T a135d04d8c3248bc954a07e5b6396fde P 7845c397230d49358ba2fad9db53f97c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7845c397230d49358ba2fad9db53f97c (127.30.32.69). New cstate: current_term: 1 leader_uuid: "7845c397230d49358ba2fad9db53f97c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:11.709249 9896 ts_tablet_manager.cc:1875] T a609406c51704d8688e0d6ccfe344ec8 P 7845c397230d49358ba2fad9db53f97c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:11.711562 9898 ts_tablet_manager.cc:1888] T a609406c51704d8688e0d6ccfe344ec8 P a47e5c56f13843018ac584914a8d4707: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:11.711648 9898 log.cc:1196] T a609406c51704d8688e0d6ccfe344ec8 P a47e5c56f13843018ac584914a8d4707: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/wal/wals/a609406c51704d8688e0d6ccfe344ec8
I20240502 20:46:11.711912 9898 ts_tablet_manager.cc:1909] T a609406c51704d8688e0d6ccfe344ec8 P a47e5c56f13843018ac584914a8d4707: Deleting consensus metadata
I20240502 20:46:11.712050 9896 ts_tablet_manager.cc:1888] T a609406c51704d8688e0d6ccfe344ec8 P 7845c397230d49358ba2fad9db53f97c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:11.712124 9896 log.cc:1196] T a609406c51704d8688e0d6ccfe344ec8 P 7845c397230d49358ba2fad9db53f97c: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/wal/wals/a609406c51704d8688e0d6ccfe344ec8
I20240502 20:46:11.712350 9897 tablet_replica.cc:330] T a609406c51704d8688e0d6ccfe344ec8 P a4bf1df7fa6b4948be502c182bfbb833: stopping tablet replica
I20240502 20:46:11.712505 9897 raft_consensus.cc:2229] T a609406c51704d8688e0d6ccfe344ec8 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:11.712615 4222 catalog_manager.cc:4830] TS a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): tablet a609406c51704d8688e0d6ccfe344ec8 (table rebalancer_dynamic_table_new_name [id=7e280a862b96486b888d929cf955cc94]) successfully deleted
I20240502 20:46:11.713058 9896 ts_tablet_manager.cc:1909] T a609406c51704d8688e0d6ccfe344ec8 P 7845c397230d49358ba2fad9db53f97c: Deleting consensus metadata
I20240502 20:46:11.713945 4637 raft_consensus.cc:1203] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 20:46:11.714305 9894 tablet_bootstrap.cc:492] T a135d04d8c3248bc954a07e5b6396fde P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:46:11.714386 9894 ts_tablet_manager.cc:1367] T a135d04d8c3248bc954a07e5b6396fde P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 20:46:11.714849 9894 raft_consensus.cc:348] T a135d04d8c3248bc954a07e5b6396fde P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:11.715020 9894 raft_consensus.cc:374] T a135d04d8c3248bc954a07e5b6396fde P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:11.715088 9894 raft_consensus.cc:729] T a135d04d8c3248bc954a07e5b6396fde P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:46:11.715260 9894 consensus_queue.cc:260] T a135d04d8c3248bc954a07e5b6396fde P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:11.716358 9894 ts_tablet_manager.cc:1398] T a135d04d8c3248bc954a07e5b6396fde P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:11.713690 9897 raft_consensus.cc:2258] T a609406c51704d8688e0d6ccfe344ec8 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:11.719005 4221 catalog_manager.cc:4830] TS 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): tablet a609406c51704d8688e0d6ccfe344ec8 (table rebalancer_dynamic_table_new_name [id=7e280a862b96486b888d929cf955cc94]) successfully deleted
I20240502 20:46:11.719262 9897 ts_tablet_manager.cc:1875] T a609406c51704d8688e0d6ccfe344ec8 P a4bf1df7fa6b4948be502c182bfbb833: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:11.721148 9897 ts_tablet_manager.cc:1888] T a609406c51704d8688e0d6ccfe344ec8 P a4bf1df7fa6b4948be502c182bfbb833: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:11.721215 9897 log.cc:1196] T a609406c51704d8688e0d6ccfe344ec8 P a4bf1df7fa6b4948be502c182bfbb833: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/wal/wals/a609406c51704d8688e0d6ccfe344ec8
I20240502 20:46:11.721413 9897 ts_tablet_manager.cc:1909] T a609406c51704d8688e0d6ccfe344ec8 P a4bf1df7fa6b4948be502c182bfbb833: Deleting consensus metadata
I20240502 20:46:11.722122 4220 catalog_manager.cc:4830] TS a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867): tablet a609406c51704d8688e0d6ccfe344ec8 (table rebalancer_dynamic_table_new_name [id=7e280a862b96486b888d929cf955cc94]) successfully deleted
I20240502 20:46:11.747845 6703 raft_consensus.cc:1203] T 93b7f7de1f8c49389c3d8c1e03c67f26 P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 20:46:11.796774 9297 raft_consensus.cc:482] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:11.796909 9297 raft_consensus.cc:504] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:11.797358 9297 leader_election.cc:288] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
I20240502 20:46:11.797765 4377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9258667ccc3e4ef8b26fe75531dd4591" candidate_uuid: "7845c397230d49358ba2fad9db53f97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2" is_pre_election: true
I20240502 20:46:11.797832 6967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9258667ccc3e4ef8b26fe75531dd4591" candidate_uuid: "7845c397230d49358ba2fad9db53f97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47e5c56f13843018ac584914a8d4707" is_pre_election: true
I20240502 20:46:11.797923 4377 raft_consensus.cc:2454] T 9258667ccc3e4ef8b26fe75531dd4591 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7845c397230d49358ba2fad9db53f97c in term 0.
I20240502 20:46:11.797951 6967 raft_consensus.cc:2454] T 9258667ccc3e4ef8b26fe75531dd4591 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7845c397230d49358ba2fad9db53f97c in term 0.
I20240502 20:46:11.798216 6639 leader_election.cc:302] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7845c397230d49358ba2fad9db53f97c, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:46:11.798383 9297 raft_consensus.cc:2781] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:11.798442 9297 raft_consensus.cc:482] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:11.798493 9297 raft_consensus.cc:3037] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:11.799324 9297 raft_consensus.cc:504] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:11.799638 9297 leader_election.cc:288] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 election: Requested vote from peers c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
I20240502 20:46:11.799969 4377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9258667ccc3e4ef8b26fe75531dd4591" candidate_uuid: "7845c397230d49358ba2fad9db53f97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2"
I20240502 20:46:11.800082 4377 raft_consensus.cc:3037] T 9258667ccc3e4ef8b26fe75531dd4591 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:11.800154 6967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9258667ccc3e4ef8b26fe75531dd4591" candidate_uuid: "7845c397230d49358ba2fad9db53f97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47e5c56f13843018ac584914a8d4707"
I20240502 20:46:11.800257 6967 raft_consensus.cc:3037] T 9258667ccc3e4ef8b26fe75531dd4591 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:11.801213 6967 raft_consensus.cc:2454] T 9258667ccc3e4ef8b26fe75531dd4591 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7845c397230d49358ba2fad9db53f97c in term 1.
I20240502 20:46:11.801213 4377 raft_consensus.cc:2454] T 9258667ccc3e4ef8b26fe75531dd4591 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7845c397230d49358ba2fad9db53f97c in term 1.
I20240502 20:46:11.801537 6639 leader_election.cc:302] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7845c397230d49358ba2fad9db53f97c, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:46:11.801731 9297 raft_consensus.cc:2781] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:11.801934 9297 raft_consensus.cc:686] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c [term 1 LEADER]: Becoming Leader. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Running, Role: LEADER
I20240502 20:46:11.802090 9297 consensus_queue.cc:237] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:11.802501 4228 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41812:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 20:46:11.804244 4227 catalog_manager.cc:5484] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7845c397230d49358ba2fad9db53f97c (127.30.32.69). New cstate: current_term: 1 leader_uuid: "7845c397230d49358ba2fad9db53f97c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:11.808336 4618 tablet_service.cc:1467] Processing CreateTablet for tablet f7553e0da1b24ff484267c2e6feefc35 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a14a0e13a5d8440899bdd3f13abdea69]), partition=
I20240502 20:46:11.808336 7079 tablet_service.cc:1467] Processing CreateTablet for tablet f7553e0da1b24ff484267c2e6feefc35 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a14a0e13a5d8440899bdd3f13abdea69]), partition=
I20240502 20:46:11.808610 4618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7553e0da1b24ff484267c2e6feefc35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:11.808614 7079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7553e0da1b24ff484267c2e6feefc35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:11.810453 9887 tablet_bootstrap.cc:492] T f7553e0da1b24ff484267c2e6feefc35 P 201d2b6485084e7dad2cbb722af94779: Bootstrap starting.
I20240502 20:46:11.810915 9903 tablet_bootstrap.cc:492] T f7553e0da1b24ff484267c2e6feefc35 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:46:11.811291 6683 tablet_service.cc:1467] Processing CreateTablet for tablet f7553e0da1b24ff484267c2e6feefc35 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a14a0e13a5d8440899bdd3f13abdea69]), partition=
I20240502 20:46:11.811456 9887 tablet_bootstrap.cc:654] T f7553e0da1b24ff484267c2e6feefc35 P 201d2b6485084e7dad2cbb722af94779: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:11.811535 6683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7553e0da1b24ff484267c2e6feefc35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:11.811776 9903 tablet_bootstrap.cc:654] T f7553e0da1b24ff484267c2e6feefc35 P 923cfab926e7443f8ffd7b21ad0a9d91: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:11.812626 9887 tablet_bootstrap.cc:492] T f7553e0da1b24ff484267c2e6feefc35 P 201d2b6485084e7dad2cbb722af94779: No bootstrap required, opened a new log
I20240502 20:46:11.812693 9887 ts_tablet_manager.cc:1367] T f7553e0da1b24ff484267c2e6feefc35 P 201d2b6485084e7dad2cbb722af94779: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:11.812912 9903 tablet_bootstrap.cc:492] T f7553e0da1b24ff484267c2e6feefc35 P 923cfab926e7443f8ffd7b21ad0a9d91: No bootstrap required, opened a new log
I20240502 20:46:11.813163 9887 raft_consensus.cc:348] T f7553e0da1b24ff484267c2e6feefc35 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:11.813253 9903 ts_tablet_manager.cc:1367] T f7553e0da1b24ff484267c2e6feefc35 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:11.813287 9887 raft_consensus.cc:374] T f7553e0da1b24ff484267c2e6feefc35 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:11.813323 9887 raft_consensus.cc:729] T f7553e0da1b24ff484267c2e6feefc35 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Initialized, Role: FOLLOWER
I20240502 20:46:11.813445 9887 consensus_queue.cc:260] T f7553e0da1b24ff484267c2e6feefc35 P 201d2b6485084e7dad2cbb722af94779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:11.813732 9887 ts_tablet_manager.cc:1398] T f7553e0da1b24ff484267c2e6feefc35 P 201d2b6485084e7dad2cbb722af94779: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:11.813722 9903 raft_consensus.cc:348] T f7553e0da1b24ff484267c2e6feefc35 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:11.813808 9263 tablet_bootstrap.cc:492] T f7553e0da1b24ff484267c2e6feefc35 P 7845c397230d49358ba2fad9db53f97c: Bootstrap starting.
I20240502 20:46:11.813848 9903 raft_consensus.cc:374] T f7553e0da1b24ff484267c2e6feefc35 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:11.813880 9903 raft_consensus.cc:729] T f7553e0da1b24ff484267c2e6feefc35 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: FOLLOWER
I20240502 20:46:11.813993 9903 consensus_queue.cc:260] T f7553e0da1b24ff484267c2e6feefc35 P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:11.814626 9263 tablet_bootstrap.cc:654] T f7553e0da1b24ff484267c2e6feefc35 P 7845c397230d49358ba2fad9db53f97c: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:11.814934 9903 ts_tablet_manager.cc:1398] T f7553e0da1b24ff484267c2e6feefc35 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:11.815920 9263 tablet_bootstrap.cc:492] T f7553e0da1b24ff484267c2e6feefc35 P 7845c397230d49358ba2fad9db53f97c: No bootstrap required, opened a new log
I20240502 20:46:11.816010 9263 ts_tablet_manager.cc:1367] T f7553e0da1b24ff484267c2e6feefc35 P 7845c397230d49358ba2fad9db53f97c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:11.816471 9263 raft_consensus.cc:348] T f7553e0da1b24ff484267c2e6feefc35 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:11.816568 9263 raft_consensus.cc:374] T f7553e0da1b24ff484267c2e6feefc35 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:11.816591 9263 raft_consensus.cc:729] T f7553e0da1b24ff484267c2e6feefc35 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Initialized, Role: FOLLOWER
I20240502 20:46:11.816713 9263 consensus_queue.cc:260] T f7553e0da1b24ff484267c2e6feefc35 P 7845c397230d49358ba2fad9db53f97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:11.816955 9263 ts_tablet_manager.cc:1398] T f7553e0da1b24ff484267c2e6feefc35 P 7845c397230d49358ba2fad9db53f97c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:11.838142 4768 raft_consensus.cc:1203] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 20:46:11.841739 7098 raft_consensus.cc:1203] T 20373b53281a4214b27f014396f7a651 P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:11.889516 4507 raft_consensus.cc:1203] T 52e302882311488ab1cac90ad03fe270 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 20:46:11.895617 8907 raft_consensus.cc:1050] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc: attempting to promote NON_VOTER 201d2b6485084e7dad2cbb722af94779 to VOTER
I20240502 20:46:11.896080 8907 consensus_queue.cc:237] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } }
I20240502 20:46:11.897564 4377 raft_consensus.cc:1261] T 41f84d5db2034af591b56e109be67934 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:11.897759 4637 raft_consensus.cc:1261] T 41f84d5db2034af591b56e109be67934 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:11.897763 7098 raft_consensus.cc:1261] T 41f84d5db2034af591b56e109be67934 P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:11.898023 6375 consensus_queue.cc:1035] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:11.898207 8146 consensus_queue.cc:1035] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:11.898823 9936 consensus_queue.cc:1035] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:11.900772 6375 raft_consensus.cc:2932] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:11.901259 4374 raft_consensus.cc:2932] T 41f84d5db2034af591b56e109be67934 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:11.903594 4227 catalog_manager.cc:5484] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: config changed from index 4 to 5, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:11.904402 4637 raft_consensus.cc:2932] T 41f84d5db2034af591b56e109be67934 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:11.904470 7098 raft_consensus.cc:2932] T 41f84d5db2034af591b56e109be67934 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:11.908372 9945 ts_tablet_manager.cc:902] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd: Initiating tablet copy from peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
I20240502 20:46:11.909245 9945 tablet_copy_client.cc:308] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.71:46841
I20240502 20:46:11.909999 6987 tablet_copy_service.cc:140] P a47e5c56f13843018ac584914a8d4707: Received BeginTabletCopySession request for tablet bcfd626ea0a04fa7b03dd0ede6b35266 from peer ed47c023452a497192f35d2f617bf9bd ({username='slave'} at 127.30.32.68:35313)
I20240502 20:46:11.910130 6987 tablet_copy_service.cc:161] P a47e5c56f13843018ac584914a8d4707: Beginning new tablet copy session on tablet bcfd626ea0a04fa7b03dd0ede6b35266 from peer ed47c023452a497192f35d2f617bf9bd at {username='slave'} at 127.30.32.68:35313: session id = ed47c023452a497192f35d2f617bf9bd-bcfd626ea0a04fa7b03dd0ede6b35266
I20240502 20:46:11.910596 4507 consensus_queue.cc:237] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } }
I20240502 20:46:11.911113 6987 tablet_copy_source_session.cc:205] T bcfd626ea0a04fa7b03dd0ede6b35266 P a47e5c56f13843018ac584914a8d4707: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:11.911737 7098 raft_consensus.cc:1261] T 41f84d5db2034af591b56e109be67934 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:11.911938 4637 raft_consensus.cc:1261] T 41f84d5db2034af591b56e109be67934 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:11.912048 9945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcfd626ea0a04fa7b03dd0ede6b35266. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:11.912846 8907 consensus_queue.cc:1035] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:11.913247 8907 consensus_queue.cc:1035] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:11.914772 9945 tablet_copy_client.cc:783] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:11.915135 9945 tablet_copy_client.cc:647] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:11.915560 8907 raft_consensus.cc:2932] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:11.915937 7098 raft_consensus.cc:2932] T 41f84d5db2034af591b56e109be67934 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:11.916386 9945 tablet_copy_client.cc:523] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:11.917639 9945 tablet_bootstrap.cc:492] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:46:11.917634 4637 raft_consensus.cc:2932] T 41f84d5db2034af591b56e109be67934 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:11.917966 4221 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 41f84d5db2034af591b56e109be67934 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:11.920312 4231 catalog_manager.cc:5484] T 41f84d5db2034af591b56e109be67934 P 201d2b6485084e7dad2cbb722af94779 reported cstate change: config changed from index 5 to 6, VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) evicted. New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:11.922751 9945 tablet_bootstrap.cc:492] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:11.923017 9945 tablet_bootstrap.cc:492] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd: Bootstrap complete.
I20240502 20:46:11.923171 9945 ts_tablet_manager.cc:1367] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20240502 20:46:11.923542 9945 raft_consensus.cc:348] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: true } }
I20240502 20:46:11.923651 9945 raft_consensus.cc:729] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: LEARNER
I20240502 20:46:11.923750 9945 consensus_queue.cc:260] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: true } }
I20240502 20:46:11.924043 9945 ts_tablet_manager.cc:1398] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:11.925062 6987 tablet_copy_service.cc:342] P a47e5c56f13843018ac584914a8d4707: Request end of tablet copy session ed47c023452a497192f35d2f617bf9bd-bcfd626ea0a04fa7b03dd0ede6b35266 received from {username='slave'} at 127.30.32.68:35313
I20240502 20:46:11.925149 6987 tablet_copy_service.cc:433] P a47e5c56f13843018ac584914a8d4707: ending tablet copy session ed47c023452a497192f35d2f617bf9bd-bcfd626ea0a04fa7b03dd0ede6b35266 on tablet bcfd626ea0a04fa7b03dd0ede6b35266 with peer ed47c023452a497192f35d2f617bf9bd
I20240502 20:46:11.925973 4354 tablet_service.cc:1514] Processing DeleteTablet for tablet 41f84d5db2034af591b56e109be67934 with delete_type TABLET_DATA_TOMBSTONED (TS c92e6e46782849479c8a0c6d82276fb2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38306
I20240502 20:46:11.926746 9956 tablet_replica.cc:330] T 41f84d5db2034af591b56e109be67934 P c92e6e46782849479c8a0c6d82276fb2: stopping tablet replica
I20240502 20:46:11.926908 9956 raft_consensus.cc:2229] T 41f84d5db2034af591b56e109be67934 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:11.927165 9956 raft_consensus.cc:2258] T 41f84d5db2034af591b56e109be67934 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:11.927907 9956 ts_tablet_manager.cc:1875] T 41f84d5db2034af591b56e109be67934 P c92e6e46782849479c8a0c6d82276fb2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:11.931540 9956 ts_tablet_manager.cc:1888] T 41f84d5db2034af591b56e109be67934 P c92e6e46782849479c8a0c6d82276fb2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:46:11.931623 9956 log.cc:1196] T 41f84d5db2034af591b56e109be67934 P c92e6e46782849479c8a0c6d82276fb2: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal/wals/41f84d5db2034af591b56e109be67934
I20240502 20:46:11.932183 4222 catalog_manager.cc:4830] TS c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827): tablet 41f84d5db2034af591b56e109be67934 (table rebalance_test_table_13 [id=c22b8adbea7b4dc49a4794a5d8f81f9b]) successfully deleted
I20240502 20:46:11.963094 7098 raft_consensus.cc:1203] T a7ea68e8581a4cd5a6212abc96dfbcde P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:11.975346 8460 raft_consensus.cc:482] T f7553e0da1b24ff484267c2e6feefc35 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:11.975459 8460 raft_consensus.cc:504] T f7553e0da1b24ff484267c2e6feefc35 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:11.975941 8460 leader_election.cc:288] T f7553e0da1b24ff484267c2e6feefc35 P 201d2b6485084e7dad2cbb722af94779 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141), 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
I20240502 20:46:11.976214 4637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7553e0da1b24ff484267c2e6feefc35" candidate_uuid: "201d2b6485084e7dad2cbb722af94779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" is_pre_election: true
I20240502 20:46:11.976372 4637 raft_consensus.cc:2454] T f7553e0da1b24ff484267c2e6feefc35 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 201d2b6485084e7dad2cbb722af94779 in term 0.
I20240502 20:46:11.976660 7032 leader_election.cc:302] T f7553e0da1b24ff484267c2e6feefc35 P 201d2b6485084e7dad2cbb722af94779 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201d2b6485084e7dad2cbb722af94779, 923cfab926e7443f8ffd7b21ad0a9d91; no voters:
I20240502 20:46:11.976872 8460 raft_consensus.cc:2781] T f7553e0da1b24ff484267c2e6feefc35 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:11.976924 8460 raft_consensus.cc:482] T f7553e0da1b24ff484267c2e6feefc35 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:11.976963 8460 raft_consensus.cc:3037] T f7553e0da1b24ff484267c2e6feefc35 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:11.978024 8460 raft_consensus.cc:504] T f7553e0da1b24ff484267c2e6feefc35 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:11.978353 8460 leader_election.cc:288] T f7553e0da1b24ff484267c2e6feefc35 P 201d2b6485084e7dad2cbb722af94779 [CANDIDATE]: Term 1 election: Requested vote from peers 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141), 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
I20240502 20:46:11.978734 4637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7553e0da1b24ff484267c2e6feefc35" candidate_uuid: "201d2b6485084e7dad2cbb722af94779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91"
I20240502 20:46:11.978847 4637 raft_consensus.cc:3037] T f7553e0da1b24ff484267c2e6feefc35 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:11.979840 4637 raft_consensus.cc:2454] T f7553e0da1b24ff484267c2e6feefc35 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 201d2b6485084e7dad2cbb722af94779 in term 1.
I20240502 20:46:11.981151 7032 leader_election.cc:302] T f7553e0da1b24ff484267c2e6feefc35 P 201d2b6485084e7dad2cbb722af94779 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201d2b6485084e7dad2cbb722af94779, 923cfab926e7443f8ffd7b21ad0a9d91; no voters:
I20240502 20:46:11.981317 8460 raft_consensus.cc:2781] T f7553e0da1b24ff484267c2e6feefc35 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:11.981397 8460 raft_consensus.cc:686] T f7553e0da1b24ff484267c2e6feefc35 P 201d2b6485084e7dad2cbb722af94779 [term 1 LEADER]: Becoming Leader. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Running, Role: LEADER
I20240502 20:46:11.981531 8460 consensus_queue.cc:237] T f7553e0da1b24ff484267c2e6feefc35 P 201d2b6485084e7dad2cbb722af94779 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:11.982087 6702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7553e0da1b24ff484267c2e6feefc35" candidate_uuid: "201d2b6485084e7dad2cbb722af94779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7845c397230d49358ba2fad9db53f97c"
I20240502 20:46:11.982216 6702 raft_consensus.cc:3037] T f7553e0da1b24ff484267c2e6feefc35 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:11.982625 6703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7553e0da1b24ff484267c2e6feefc35" candidate_uuid: "201d2b6485084e7dad2cbb722af94779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7845c397230d49358ba2fad9db53f97c" is_pre_election: true
I20240502 20:46:11.981683 10004 ts_tablet_manager.cc:902] T e7e0539e60064d97a7eb0eacd9df065e P a4bf1df7fa6b4948be502c182bfbb833: Initiating tablet copy from peer 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:46:11.983251 6702 raft_consensus.cc:2454] T f7553e0da1b24ff484267c2e6feefc35 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 201d2b6485084e7dad2cbb722af94779 in term 1.
I20240502 20:46:11.983784 4227 catalog_manager.cc:5484] T f7553e0da1b24ff484267c2e6feefc35 P 201d2b6485084e7dad2cbb722af94779 reported cstate change: term changed from 0 to 1, leader changed from <none> to 201d2b6485084e7dad2cbb722af94779 (127.30.32.72). New cstate: current_term: 1 leader_uuid: "201d2b6485084e7dad2cbb722af94779" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } health_report { overall_health: HEALTHY } } }
I20240502 20:46:11.984126 10004 tablet_copy_client.cc:308] T e7e0539e60064d97a7eb0eacd9df065e P a4bf1df7fa6b4948be502c182bfbb833: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.67:42141
I20240502 20:46:11.984563 4658 tablet_copy_service.cc:140] P 923cfab926e7443f8ffd7b21ad0a9d91: Received BeginTabletCopySession request for tablet e7e0539e60064d97a7eb0eacd9df065e from peer a4bf1df7fa6b4948be502c182bfbb833 ({username='slave'} at 127.30.32.70:32901)
I20240502 20:46:11.984658 4658 tablet_copy_service.cc:161] P 923cfab926e7443f8ffd7b21ad0a9d91: Beginning new tablet copy session on tablet e7e0539e60064d97a7eb0eacd9df065e from peer a4bf1df7fa6b4948be502c182bfbb833 at {username='slave'} at 127.30.32.70:32901: session id = a4bf1df7fa6b4948be502c182bfbb833-e7e0539e60064d97a7eb0eacd9df065e
I20240502 20:46:11.985620 4658 tablet_copy_source_session.cc:205] T e7e0539e60064d97a7eb0eacd9df065e P 923cfab926e7443f8ffd7b21ad0a9d91: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:11.986919 10004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7e0539e60064d97a7eb0eacd9df065e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:11.987706 10015 ts_tablet_manager.cc:902] T c9d25657e9a14b35b2f6e8d82f4533c7 P 7845c397230d49358ba2fad9db53f97c: Initiating tablet copy from peer c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:46:11.988166 10015 tablet_copy_client.cc:308] T c9d25657e9a14b35b2f6e8d82f4533c7 P 7845c397230d49358ba2fad9db53f97c: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.65:33827
I20240502 20:46:11.988741 4397 tablet_copy_service.cc:140] P c92e6e46782849479c8a0c6d82276fb2: Received BeginTabletCopySession request for tablet c9d25657e9a14b35b2f6e8d82f4533c7 from peer 7845c397230d49358ba2fad9db53f97c ({username='slave'} at 127.30.32.69:38945)
I20240502 20:46:11.988857 4397 tablet_copy_service.cc:161] P c92e6e46782849479c8a0c6d82276fb2: Beginning new tablet copy session on tablet c9d25657e9a14b35b2f6e8d82f4533c7 from peer 7845c397230d49358ba2fad9db53f97c at {username='slave'} at 127.30.32.69:38945: session id = 7845c397230d49358ba2fad9db53f97c-c9d25657e9a14b35b2f6e8d82f4533c7
I20240502 20:46:11.989651 10004 tablet_copy_client.cc:783] T e7e0539e60064d97a7eb0eacd9df065e P a4bf1df7fa6b4948be502c182bfbb833: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:11.989859 10004 tablet_copy_client.cc:647] T e7e0539e60064d97a7eb0eacd9df065e P a4bf1df7fa6b4948be502c182bfbb833: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:11.989894 4397 tablet_copy_source_session.cc:205] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:11.990757 10015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9d25657e9a14b35b2f6e8d82f4533c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:11.991187 10004 tablet_copy_client.cc:523] T e7e0539e60064d97a7eb0eacd9df065e P a4bf1df7fa6b4948be502c182bfbb833: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:11.992520 10004 tablet_bootstrap.cc:492] T e7e0539e60064d97a7eb0eacd9df065e P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap starting.
I20240502 20:46:11.993083 10015 tablet_copy_client.cc:783] T c9d25657e9a14b35b2f6e8d82f4533c7 P 7845c397230d49358ba2fad9db53f97c: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:11.993245 10015 tablet_copy_client.cc:647] T c9d25657e9a14b35b2f6e8d82f4533c7 P 7845c397230d49358ba2fad9db53f97c: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:11.995102 10015 tablet_copy_client.cc:523] T c9d25657e9a14b35b2f6e8d82f4533c7 P 7845c397230d49358ba2fad9db53f97c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:11.996388 10015 tablet_bootstrap.cc:492] T c9d25657e9a14b35b2f6e8d82f4533c7 P 7845c397230d49358ba2fad9db53f97c: Bootstrap starting.
I20240502 20:46:11.999399 6967 raft_consensus.cc:1203] T d37fab152a8a44868d3160ec3057bb98 P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 20:46:12.003955 9042 raft_consensus.cc:1050] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91: attempting to promote NON_VOTER 7845c397230d49358ba2fad9db53f97c to VOTER
I20240502 20:46:12.004395 9042 consensus_queue.cc:237] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } }
I20240502 20:46:12.005661 6702 raft_consensus.cc:1261] T 02302fc79dc04b0396041c600c2f4bef P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:12.006414 4507 raft_consensus.cc:1261] T 02302fc79dc04b0396041c600c2f4bef P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:12.007017 9041 consensus_queue.cc:1035] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:12.007817 4374 raft_consensus.cc:1261] T 02302fc79dc04b0396041c600c2f4bef P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:12.008445 7017 consensus_queue.cc:1035] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:12.008770 7017 consensus_queue.cc:1035] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:12.012153 9041 raft_consensus.cc:2932] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:12.015265 4227 catalog_manager.cc:5484] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index 4 to 5, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:12.015411 4374 raft_consensus.cc:2932] T 02302fc79dc04b0396041c600c2f4bef P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:12.017676 6702 raft_consensus.cc:2932] T 02302fc79dc04b0396041c600c2f4bef P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:12.019379 4507 raft_consensus.cc:2932] T 02302fc79dc04b0396041c600c2f4bef P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:12.026023 4637 consensus_queue.cc:237] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } }
I20240502 20:46:12.027433 6702 raft_consensus.cc:1261] T 02302fc79dc04b0396041c600c2f4bef P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:12.027560 4374 raft_consensus.cc:1261] T 02302fc79dc04b0396041c600c2f4bef P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:12.027911 9043 consensus_queue.cc:1035] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:12.028080 9042 consensus_queue.cc:1035] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:12.029528 8236 raft_consensus.cc:2932] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:12.029946 6702 raft_consensus.cc:2932] T 02302fc79dc04b0396041c600c2f4bef P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:12.031330 4219 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 02302fc79dc04b0396041c600c2f4bef with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:12.031870 4374 raft_consensus.cc:2932] T 02302fc79dc04b0396041c600c2f4bef P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:12.032383 10015 tablet_bootstrap.cc:492] T c9d25657e9a14b35b2f6e8d82f4533c7 P 7845c397230d49358ba2fad9db53f97c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:12.032461 4230 catalog_manager.cc:5484] T 02302fc79dc04b0396041c600c2f4bef P 7845c397230d49358ba2fad9db53f97c reported cstate change: config changed from index 5 to 6, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:12.032670 10015 tablet_bootstrap.cc:492] T c9d25657e9a14b35b2f6e8d82f4533c7 P 7845c397230d49358ba2fad9db53f97c: Bootstrap complete.
I20240502 20:46:12.032832 10015 ts_tablet_manager.cc:1367] T c9d25657e9a14b35b2f6e8d82f4533c7 P 7845c397230d49358ba2fad9db53f97c: Time spent bootstrapping tablet: real 0.036s user 0.018s sys 0.008s
I20240502 20:46:12.033313 10015 raft_consensus.cc:348] T c9d25657e9a14b35b2f6e8d82f4533c7 P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } }
I20240502 20:46:12.033504 10015 raft_consensus.cc:729] T c9d25657e9a14b35b2f6e8d82f4533c7 P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Initialized, Role: LEARNER
I20240502 20:46:12.033663 10015 consensus_queue.cc:260] T c9d25657e9a14b35b2f6e8d82f4533c7 P 7845c397230d49358ba2fad9db53f97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } }
I20240502 20:46:12.034036 10015 ts_tablet_manager.cc:1398] T c9d25657e9a14b35b2f6e8d82f4533c7 P 7845c397230d49358ba2fad9db53f97c: Time spent starting tablet: real 0.001s user 0.005s sys 0.000s
I20240502 20:46:12.034564 4397 tablet_copy_service.cc:342] P c92e6e46782849479c8a0c6d82276fb2: Request end of tablet copy session 7845c397230d49358ba2fad9db53f97c-c9d25657e9a14b35b2f6e8d82f4533c7 received from {username='slave'} at 127.30.32.69:38945
I20240502 20:46:12.034636 4397 tablet_copy_service.cc:433] P c92e6e46782849479c8a0c6d82276fb2: ending tablet copy session 7845c397230d49358ba2fad9db53f97c-c9d25657e9a14b35b2f6e8d82f4533c7 on tablet c9d25657e9a14b35b2f6e8d82f4533c7 with peer 7845c397230d49358ba2fad9db53f97c
I20240502 20:46:12.035705 10004 tablet_bootstrap.cc:492] T e7e0539e60064d97a7eb0eacd9df065e P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:12.036010 10004 tablet_bootstrap.cc:492] T e7e0539e60064d97a7eb0eacd9df065e P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap complete.
I20240502 20:46:12.036170 10004 ts_tablet_manager.cc:1367] T e7e0539e60064d97a7eb0eacd9df065e P a4bf1df7fa6b4948be502c182bfbb833: Time spent bootstrapping tablet: real 0.044s user 0.018s sys 0.004s
I20240502 20:46:12.036540 4487 tablet_service.cc:1514] Processing DeleteTablet for tablet 02302fc79dc04b0396041c600c2f4bef with delete_type TABLET_DATA_TOMBSTONED (TS c3d26eafd51547acb780d4aac5ebe7cc not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48890
I20240502 20:46:12.036559 10004 raft_consensus.cc:348] T e7e0539e60064d97a7eb0eacd9df065e P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } }
I20240502 20:46:12.036662 10004 raft_consensus.cc:729] T e7e0539e60064d97a7eb0eacd9df065e P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Initialized, Role: LEARNER
I20240502 20:46:12.036860 10045 tablet_replica.cc:330] T 02302fc79dc04b0396041c600c2f4bef P c3d26eafd51547acb780d4aac5ebe7cc: stopping tablet replica
I20240502 20:46:12.037020 10045 raft_consensus.cc:2229] T 02302fc79dc04b0396041c600c2f4bef P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:12.038259 10045 raft_consensus.cc:2258] T 02302fc79dc04b0396041c600c2f4bef P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:12.038914 10045 ts_tablet_manager.cc:1875] T 02302fc79dc04b0396041c600c2f4bef P c3d26eafd51547acb780d4aac5ebe7cc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:12.041131 10045 ts_tablet_manager.cc:1888] T 02302fc79dc04b0396041c600c2f4bef P c3d26eafd51547acb780d4aac5ebe7cc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:46:12.041214 10045 log.cc:1196] T 02302fc79dc04b0396041c600c2f4bef P c3d26eafd51547acb780d4aac5ebe7cc: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal/wals/02302fc79dc04b0396041c600c2f4bef
I20240502 20:46:12.041741 4221 catalog_manager.cc:4830] TS c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097): tablet 02302fc79dc04b0396041c600c2f4bef (table rebalance_test_table_19 [id=46a52eb51c614964a91ee65db96cf854]) successfully deleted
I20240502 20:46:12.045183 10004 consensus_queue.cc:260] T e7e0539e60064d97a7eb0eacd9df065e P a4bf1df7fa6b4948be502c182bfbb833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } }
I20240502 20:46:12.045639 10004 ts_tablet_manager.cc:1398] T e7e0539e60064d97a7eb0eacd9df065e P a4bf1df7fa6b4948be502c182bfbb833: Time spent starting tablet: real 0.009s user 0.003s sys 0.001s
I20240502 20:46:12.046273 4658 tablet_copy_service.cc:342] P 923cfab926e7443f8ffd7b21ad0a9d91: Request end of tablet copy session a4bf1df7fa6b4948be502c182bfbb833-e7e0539e60064d97a7eb0eacd9df065e received from {username='slave'} at 127.30.32.70:32901
I20240502 20:46:12.046358 4658 tablet_copy_service.cc:433] P 923cfab926e7443f8ffd7b21ad0a9d91: ending tablet copy session a4bf1df7fa6b4948be502c182bfbb833-e7e0539e60064d97a7eb0eacd9df065e on tablet e7e0539e60064d97a7eb0eacd9df065e with peer a4bf1df7fa6b4948be502c182bfbb833
I20240502 20:46:12.053126 6967 raft_consensus.cc:1261] T 8565ebbd21dd4f708a0254d1ed2d6670 P a47e5c56f13843018ac584914a8d4707 [term 2 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 20:46:12.053685 8146 consensus_queue.cc:1035] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:12.070132 6138 raft_consensus.cc:482] T a135d04d8c3248bc954a07e5b6396fde P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:12.070230 6138 raft_consensus.cc:504] T a135d04d8c3248bc954a07e5b6396fde P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:12.070602 6138 leader_election.cc:288] T a135d04d8c3248bc954a07e5b6396fde P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
I20240502 20:46:12.070933 6702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a135d04d8c3248bc954a07e5b6396fde" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7845c397230d49358ba2fad9db53f97c" is_pre_election: true
I20240502 20:46:12.071036 7098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a135d04d8c3248bc954a07e5b6396fde" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201d2b6485084e7dad2cbb722af94779" is_pre_election: true
I20240502 20:46:12.071192 7098 raft_consensus.cc:2379] T a135d04d8c3248bc954a07e5b6396fde P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c92e6e46782849479c8a0c6d82276fb2 in current term 1: Already voted for candidate 7845c397230d49358ba2fad9db53f97c in this term.
I20240502 20:46:12.071451 4310 leader_election.cc:302] T a135d04d8c3248bc954a07e5b6396fde P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2; no voters: 201d2b6485084e7dad2cbb722af94779, 7845c397230d49358ba2fad9db53f97c
I20240502 20:46:12.071650 6138 raft_consensus.cc:3037] T a135d04d8c3248bc954a07e5b6396fde P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:12.072664 6138 raft_consensus.cc:2726] T a135d04d8c3248bc954a07e5b6396fde P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 20:46:12.084510 9042 raft_consensus.cc:1050] T d6c2626b3ee7493c8a349f08e17082b5 P 923cfab926e7443f8ffd7b21ad0a9d91: attempting to promote NON_VOTER c3d26eafd51547acb780d4aac5ebe7cc to VOTER
I20240502 20:46:12.085003 9042 consensus_queue.cc:237] T d6c2626b3ee7493c8a349f08e17082b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } }
I20240502 20:46:12.086665 6967 raft_consensus.cc:1261] T d6c2626b3ee7493c8a349f08e17082b5 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:12.086751 4507 raft_consensus.cc:1261] T d6c2626b3ee7493c8a349f08e17082b5 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEARNER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:12.086994 6835 raft_consensus.cc:1261] T d6c2626b3ee7493c8a349f08e17082b5 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:12.087095 9041 consensus_queue.cc:1035] T d6c2626b3ee7493c8a349f08e17082b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:46:12.087267 8236 consensus_queue.cc:1035] T d6c2626b3ee7493c8a349f08e17082b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:46:12.087473 7017 consensus_queue.cc:1035] T d6c2626b3ee7493c8a349f08e17082b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:46:12.087466 9297 consensus_queue.cc:1035] T a135d04d8c3248bc954a07e5b6396fde P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:12.090721 9041 raft_consensus.cc:2932] T d6c2626b3ee7493c8a349f08e17082b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } } }
I20240502 20:46:12.091995 6967 raft_consensus.cc:2932] T d6c2626b3ee7493c8a349f08e17082b5 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } } }
I20240502 20:46:12.093140 4507 raft_consensus.cc:2932] T d6c2626b3ee7493c8a349f08e17082b5 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } } }
I20240502 20:46:12.094224 9105 consensus_queue.cc:1035] T a135d04d8c3248bc954a07e5b6396fde P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:12.094766 4227 catalog_manager.cc:5484] T d6c2626b3ee7493c8a349f08e17082b5 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index 2 to 3, c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:12.101759 6835 raft_consensus.cc:2932] T d6c2626b3ee7493c8a349f08e17082b5 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } } }
I20240502 20:46:12.103562 7017 raft_consensus.cc:1050] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91: attempting to promote NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 to VOTER
I20240502 20:46:12.103616 4637 consensus_queue.cc:237] T d6c2626b3ee7493c8a349f08e17082b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } }
I20240502 20:46:12.103962 7017 consensus_queue.cc:237] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } }
I20240502 20:46:12.105360 4768 raft_consensus.cc:1261] T 28da7347f982468281570ffa7503a854 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:12.105603 4638 raft_consensus.cc:1261] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 2 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 20:46:12.105793 4231 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 20:46:12.106010 8146 consensus_queue.cc:1035] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:12.106199 4507 raft_consensus.cc:1261] T 28da7347f982468281570ffa7503a854 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:12.106423 9775 raft_consensus.cc:482] T a999f994b78a4b33a95ee7d5407cba20 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:12.106511 9775 raft_consensus.cc:504] T a999f994b78a4b33a95ee7d5407cba20 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:12.106540 9042 consensus_queue.cc:1035] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:12.106885 9775 leader_election.cc:288] T a999f994b78a4b33a95ee7d5407cba20 P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867)
I20240502 20:46:12.107203 6702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a999f994b78a4b33a95ee7d5407cba20" candidate_uuid: "a47e5c56f13843018ac584914a8d4707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7845c397230d49358ba2fad9db53f97c" is_pre_election: true
I20240502 20:46:12.107340 6702 raft_consensus.cc:2454] T a999f994b78a4b33a95ee7d5407cba20 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a47e5c56f13843018ac584914a8d4707 in term 0.
I20240502 20:46:12.107645 6902 leader_election.cc:302] T a999f994b78a4b33a95ee7d5407cba20 P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7845c397230d49358ba2fad9db53f97c, a47e5c56f13843018ac584914a8d4707; no voters:
I20240502 20:46:12.107810 9775 raft_consensus.cc:2781] T a999f994b78a4b33a95ee7d5407cba20 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:12.107880 9775 raft_consensus.cc:482] T a999f994b78a4b33a95ee7d5407cba20 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:12.107920 9775 raft_consensus.cc:3037] T a999f994b78a4b33a95ee7d5407cba20 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:12.108268 6835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a999f994b78a4b33a95ee7d5407cba20" candidate_uuid: "a47e5c56f13843018ac584914a8d4707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833" is_pre_election: true
I20240502 20:46:12.108387 6835 raft_consensus.cc:2454] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a47e5c56f13843018ac584914a8d4707 in term 0.
I20240502 20:46:12.109054 9775 raft_consensus.cc:504] T a999f994b78a4b33a95ee7d5407cba20 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:12.109211 6834 raft_consensus.cc:1261] T 28da7347f982468281570ffa7503a854 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEARNER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:12.109414 9775 leader_election.cc:288] T a999f994b78a4b33a95ee7d5407cba20 P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 1 election: Requested vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867)
I20240502 20:46:12.110498 6702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a999f994b78a4b33a95ee7d5407cba20" candidate_uuid: "a47e5c56f13843018ac584914a8d4707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7845c397230d49358ba2fad9db53f97c"
I20240502 20:46:12.110627 6702 raft_consensus.cc:3037] T a999f994b78a4b33a95ee7d5407cba20 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:12.111789 6702 raft_consensus.cc:2454] T a999f994b78a4b33a95ee7d5407cba20 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a47e5c56f13843018ac584914a8d4707 in term 1.
I20240502 20:46:12.111811 6703 raft_consensus.cc:1261] T f7553e0da1b24ff484267c2e6feefc35 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Refusing update from remote peer 201d2b6485084e7dad2cbb722af94779: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:12.112016 4638 raft_consensus.cc:1261] T f7553e0da1b24ff484267c2e6feefc35 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer 201d2b6485084e7dad2cbb722af94779: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:12.112238 9413 consensus_queue.cc:1035] T f7553e0da1b24ff484267c2e6feefc35 P 201d2b6485084e7dad2cbb722af94779 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:12.112429 8460 consensus_queue.cc:1035] T f7553e0da1b24ff484267c2e6feefc35 P 201d2b6485084e7dad2cbb722af94779 [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:12.112777 4507 raft_consensus.cc:1261] T d6c2626b3ee7493c8a349f08e17082b5 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:12.114069 9043 consensus_queue.cc:1035] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:12.115734 10095 tablet.cc:1722] T f7553e0da1b24ff484267c2e6feefc35 P 7845c397230d49358ba2fad9db53f97c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 20:46:12.115911 4231 catalog_manager.cc:5576] TS 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141) does not have the latest schema for tablet f7553e0da1b24ff484267c2e6feefc35 (table rebalancer_dynamic_table [id=a14a0e13a5d8440899bdd3f13abdea69]). Expected version 1 got 0
I20240502 20:46:12.116977 10095 tablet.cc:1620] T f7553e0da1b24ff484267c2e6feefc35 P 7845c397230d49358ba2fad9db53f97c: MemRowSet was empty: no flush needed.
I20240502 20:46:12.117735 4228 catalog_manager.cc:5576] TS 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643) does not have the latest schema for tablet f7553e0da1b24ff484267c2e6feefc35 (table rebalancer_dynamic_table [id=a14a0e13a5d8440899bdd3f13abdea69]). Expected version 1 got 0
I20240502 20:46:12.119642 6835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a999f994b78a4b33a95ee7d5407cba20" candidate_uuid: "a47e5c56f13843018ac584914a8d4707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833"
I20240502 20:46:12.119791 6835 raft_consensus.cc:3037] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:12.120277 6834 raft_consensus.cc:1261] T d6c2626b3ee7493c8a349f08e17082b5 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:12.120486 9042 consensus_queue.cc:1035] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:12.120622 6835 raft_consensus.cc:2454] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a47e5c56f13843018ac584914a8d4707 in term 1.
I20240502 20:46:12.121054 6901 leader_election.cc:302] T a999f994b78a4b33a95ee7d5407cba20 P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a47e5c56f13843018ac584914a8d4707, a4bf1df7fa6b4948be502c182bfbb833; no voters:
I20240502 20:46:12.121290 9775 raft_consensus.cc:2781] T a999f994b78a4b33a95ee7d5407cba20 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:12.121387 9775 raft_consensus.cc:686] T a999f994b78a4b33a95ee7d5407cba20 P a47e5c56f13843018ac584914a8d4707 [term 1 LEADER]: Becoming Leader. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Running, Role: LEADER
I20240502 20:46:12.121363 9042 consensus_queue.cc:1035] T d6c2626b3ee7493c8a349f08e17082b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:12.121524 9775 consensus_queue.cc:237] T a999f994b78a4b33a95ee7d5407cba20 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:12.122589 8236 consensus_queue.cc:1035] T d6c2626b3ee7493c8a349f08e17082b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:12.124009 10094 tablet.cc:1722] T f7553e0da1b24ff484267c2e6feefc35 P 201d2b6485084e7dad2cbb722af94779: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 20:46:12.124146 4228 catalog_manager.cc:5484] T a999f994b78a4b33a95ee7d5407cba20 P a47e5c56f13843018ac584914a8d4707 reported cstate change: term changed from 0 to 1, leader changed from <none> to a47e5c56f13843018ac584914a8d4707 (127.30.32.71). New cstate: current_term: 1 leader_uuid: "a47e5c56f13843018ac584914a8d4707" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:12.124425 10094 tablet.cc:1620] T f7553e0da1b24ff484267c2e6feefc35 P 201d2b6485084e7dad2cbb722af94779: MemRowSet was empty: no flush needed.
I20240502 20:46:12.127782 6703 raft_consensus.cc:1261] T 8565ebbd21dd4f708a0254d1ed2d6670 P 7845c397230d49358ba2fad9db53f97c [term 2 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 20:46:12.128058 8236 raft_consensus.cc:2932] T d6c2626b3ee7493c8a349f08e17082b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } } }
I20240502 20:46:12.128249 8907 consensus_queue.cc:1035] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:12.129626 4231 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: rebalancer_dynamic_table [id=a14a0e13a5d8440899bdd3f13abdea69] alter complete (version 1)
I20240502 20:46:12.129736 4219 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d6c2626b3ee7493c8a349f08e17082b5 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:12.132966 4231 catalog_manager.cc:5484] T d6c2626b3ee7493c8a349f08e17082b5 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index 3 to 4, VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) evicted. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:12.134084 7017 raft_consensus.cc:2932] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:12.135085 10107 tablet.cc:1722] T f7553e0da1b24ff484267c2e6feefc35 P 923cfab926e7443f8ffd7b21ad0a9d91: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 20:46:12.135023 4508 raft_consensus.cc:2932] T d6c2626b3ee7493c8a349f08e17082b5 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } } }
I20240502 20:46:12.137080 6947 tablet_service.cc:1514] Processing DeleteTablet for tablet d6c2626b3ee7493c8a349f08e17082b5 with delete_type TABLET_DATA_TOMBSTONED (TS a47e5c56f13843018ac584914a8d4707 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45156
I20240502 20:46:12.137320 9898 tablet_replica.cc:330] T d6c2626b3ee7493c8a349f08e17082b5 P a47e5c56f13843018ac584914a8d4707: stopping tablet replica
I20240502 20:46:12.137462 9898 raft_consensus.cc:2229] T d6c2626b3ee7493c8a349f08e17082b5 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:12.137332 4768 raft_consensus.cc:2932] T 28da7347f982468281570ffa7503a854 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:12.137590 9898 raft_consensus.cc:2258] T d6c2626b3ee7493c8a349f08e17082b5 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:12.137521 6831 raft_consensus.cc:2932] T d6c2626b3ee7493c8a349f08e17082b5 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } } }
I20240502 20:46:12.137981 6835 raft_consensus.cc:2932] T 28da7347f982468281570ffa7503a854 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:12.135426 10107 tablet.cc:1620] T f7553e0da1b24ff484267c2e6feefc35 P 923cfab926e7443f8ffd7b21ad0a9d91: MemRowSet was empty: no flush needed.
I20240502 20:46:12.140820 4231 catalog_manager.cc:5484] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index 4 to 5, a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:12.140946 9898 ts_tablet_manager.cc:1875] T d6c2626b3ee7493c8a349f08e17082b5 P a47e5c56f13843018ac584914a8d4707: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:12.141880 4507 raft_consensus.cc:2932] T 28da7347f982468281570ffa7503a854 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:12.143276 9898 ts_tablet_manager.cc:1888] T d6c2626b3ee7493c8a349f08e17082b5 P a47e5c56f13843018ac584914a8d4707: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 20:46:12.143373 9898 log.cc:1196] T d6c2626b3ee7493c8a349f08e17082b5 P a47e5c56f13843018ac584914a8d4707: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/wal/wals/d6c2626b3ee7493c8a349f08e17082b5
I20240502 20:46:12.143990 4222 catalog_manager.cc:4830] TS a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): tablet d6c2626b3ee7493c8a349f08e17082b5 (table rebalancer_extra_table_0 [id=09a2b90df269425ba3ab97f05a1fc119]) successfully deleted
I20240502 20:46:12.146093 9043 raft_consensus.cc:1050] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91: attempting to promote NON_VOTER a47e5c56f13843018ac584914a8d4707 to VOTER
I20240502 20:46:12.146485 9043 consensus_queue.cc:237] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } }
I20240502 20:46:12.147783 6967 raft_consensus.cc:1261] T 00eb91e624554baf918e96b098019be8 P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:12.147955 4638 consensus_queue.cc:237] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } }
I20240502 20:46:12.148131 9043 consensus_queue.cc:1035] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:12.148222 4768 raft_consensus.cc:1261] T 00eb91e624554baf918e96b098019be8 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:12.148682 9043 consensus_queue.cc:1035] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:12.149173 4507 raft_consensus.cc:1261] T 28da7347f982468281570ffa7503a854 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:12.149231 6703 raft_consensus.cc:1261] T 00eb91e624554baf918e96b098019be8 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:12.149821 8236 consensus_queue.cc:1035] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:12.150132 8236 consensus_queue.cc:1035] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:12.151779 6835 raft_consensus.cc:1261] T 28da7347f982468281570ffa7503a854 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:12.152825 7017 consensus_queue.cc:1035] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:12.154800 9042 raft_consensus.cc:2932] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:12.154976 6835 raft_consensus.cc:2932] T 28da7347f982468281570ffa7503a854 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:12.157717 4219 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 28da7347f982468281570ffa7503a854 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:12.158057 4508 raft_consensus.cc:2932] T 28da7347f982468281570ffa7503a854 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:12.157496 4230 catalog_manager.cc:5484] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index 5 to 6, VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) evicted. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:12.161610 4750 tablet_service.cc:1514] Processing DeleteTablet for tablet 28da7347f982468281570ffa7503a854 with delete_type TABLET_DATA_TOMBSTONED (TS ed47c023452a497192f35d2f617bf9bd not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38526
I20240502 20:46:12.161664 9041 raft_consensus.cc:2932] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:12.162765 10117 tablet_replica.cc:330] T 28da7347f982468281570ffa7503a854 P ed47c023452a497192f35d2f617bf9bd: stopping tablet replica
I20240502 20:46:12.162932 10117 raft_consensus.cc:2229] T 28da7347f982468281570ffa7503a854 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:12.163115 10117 raft_consensus.cc:2258] T 28da7347f982468281570ffa7503a854 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:12.163586 6967 raft_consensus.cc:2932] T 00eb91e624554baf918e96b098019be8 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:12.163635 4768 raft_consensus.cc:2932] T 00eb91e624554baf918e96b098019be8 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:12.165249 4228 catalog_manager.cc:5484] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index 7 to 8, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:12.165690 10117 ts_tablet_manager.cc:1875] T 28da7347f982468281570ffa7503a854 P ed47c023452a497192f35d2f617bf9bd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:12.166851 6703 raft_consensus.cc:2932] T 00eb91e624554baf918e96b098019be8 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:12.168562 4638 consensus_queue.cc:237] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } }
I20240502 20:46:12.169835 6967 raft_consensus.cc:1261] T 00eb91e624554baf918e96b098019be8 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:46:12.170331 7017 consensus_queue.cc:1035] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:12.172591 6703 raft_consensus.cc:1261] T 00eb91e624554baf918e96b098019be8 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:46:12.173015 7017 consensus_queue.cc:1035] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:12.173602 10117 ts_tablet_manager.cc:1888] T 28da7347f982468281570ffa7503a854 P ed47c023452a497192f35d2f617bf9bd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:46:12.173729 10117 log.cc:1196] T 28da7347f982468281570ffa7503a854 P ed47c023452a497192f35d2f617bf9bd: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal/wals/28da7347f982468281570ffa7503a854
I20240502 20:46:12.174324 4222 catalog_manager.cc:4830] TS ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193): tablet 28da7347f982468281570ffa7503a854 (table rebalance_test_table_1 [id=3a26d60c6de746e891d970c2ce9c55cc]) successfully deleted
I20240502 20:46:12.174800 7017 raft_consensus.cc:2932] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:12.177136 4219 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 00eb91e624554baf918e96b098019be8 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:12.177793 6703 raft_consensus.cc:2932] T 00eb91e624554baf918e96b098019be8 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:12.181311 6967 raft_consensus.cc:2932] T 00eb91e624554baf918e96b098019be8 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:12.183259 4228 catalog_manager.cc:5484] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index 8 to 9, VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) evicted. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:12.186102 4750 tablet_service.cc:1514] Processing DeleteTablet for tablet 00eb91e624554baf918e96b098019be8 with delete_type TABLET_DATA_TOMBSTONED (TS ed47c023452a497192f35d2f617bf9bd not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:38526
I20240502 20:46:12.188311 10117 tablet_replica.cc:330] T 00eb91e624554baf918e96b098019be8 P ed47c023452a497192f35d2f617bf9bd: stopping tablet replica
I20240502 20:46:12.188501 10117 raft_consensus.cc:2229] T 00eb91e624554baf918e96b098019be8 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:12.188724 10117 raft_consensus.cc:2258] T 00eb91e624554baf918e96b098019be8 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:12.189481 10117 ts_tablet_manager.cc:1875] T 00eb91e624554baf918e96b098019be8 P ed47c023452a497192f35d2f617bf9bd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:12.191326 10117 ts_tablet_manager.cc:1888] T 00eb91e624554baf918e96b098019be8 P ed47c023452a497192f35d2f617bf9bd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 20:46:12.191397 10117 log.cc:1196] T 00eb91e624554baf918e96b098019be8 P ed47c023452a497192f35d2f617bf9bd: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal/wals/00eb91e624554baf918e96b098019be8
I20240502 20:46:12.191954 4222 catalog_manager.cc:4830] TS ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193): tablet 00eb91e624554baf918e96b098019be8 (table rebalance_test_table_3 [id=54d5208bcf1c43998e3ac81026ea9fbe]) successfully deleted
I20240502 20:46:12.199919 9155 raft_consensus.cc:1050] T 8602022a45e1482ebf8dab9930ce8209 P a47e5c56f13843018ac584914a8d4707: attempting to promote NON_VOTER 923cfab926e7443f8ffd7b21ad0a9d91 to VOTER
I20240502 20:46:12.200381 9155 consensus_queue.cc:237] T 8602022a45e1482ebf8dab9930ce8209 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } }
I20240502 20:46:12.201745 4638 raft_consensus.cc:1261] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEARNER]: Refusing update from remote peer a47e5c56f13843018ac584914a8d4707: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:12.201970 6703 raft_consensus.cc:1261] T 8602022a45e1482ebf8dab9930ce8209 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Refusing update from remote peer a47e5c56f13843018ac584914a8d4707: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:12.202123 6835 raft_consensus.cc:1261] T 8602022a45e1482ebf8dab9930ce8209 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Refusing update from remote peer a47e5c56f13843018ac584914a8d4707: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:12.202556 9155 consensus_queue.cc:1035] T 8602022a45e1482ebf8dab9930ce8209 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:46:12.202745 9775 consensus_queue.cc:1035] T 8602022a45e1482ebf8dab9930ce8209 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:46:12.203214 10138 consensus_queue.cc:1035] T 8602022a45e1482ebf8dab9930ce8209 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:46:12.205403 9155 raft_consensus.cc:2932] T 8602022a45e1482ebf8dab9930ce8209 P a47e5c56f13843018ac584914a8d4707 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:12.206430 6835 raft_consensus.cc:2932] T 8602022a45e1482ebf8dab9930ce8209 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:12.207988 6703 raft_consensus.cc:2932] T 8602022a45e1482ebf8dab9930ce8209 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:12.208420 4228 catalog_manager.cc:5484] T 8602022a45e1482ebf8dab9930ce8209 P a47e5c56f13843018ac584914a8d4707 reported cstate change: config changed from index 2 to 3, 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a47e5c56f13843018ac584914a8d4707" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:12.214227 4637 raft_consensus.cc:2932] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:12.216006 6967 consensus_queue.cc:237] T 8602022a45e1482ebf8dab9930ce8209 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } }
I20240502 20:46:12.217227 6703 raft_consensus.cc:1261] T 8602022a45e1482ebf8dab9930ce8209 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Refusing update from remote peer a47e5c56f13843018ac584914a8d4707: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:12.217687 9775 consensus_queue.cc:1035] T 8602022a45e1482ebf8dab9930ce8209 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:12.217962 4637 raft_consensus.cc:1261] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer a47e5c56f13843018ac584914a8d4707: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:12.218374 9775 consensus_queue.cc:1035] T 8602022a45e1482ebf8dab9930ce8209 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:12.219327 9775 raft_consensus.cc:2932] T 8602022a45e1482ebf8dab9930ce8209 P a47e5c56f13843018ac584914a8d4707 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:12.220644 4222 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8602022a45e1482ebf8dab9930ce8209 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:12.221783 4638 consensus_queue.cc:237] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } }
I20240502 20:46:12.222183 4233 catalog_manager.cc:5484] T 8602022a45e1482ebf8dab9930ce8209 P a47e5c56f13843018ac584914a8d4707 reported cstate change: config changed from index 3 to 4, VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) evicted. New cstate: current_term: 1 leader_uuid: "a47e5c56f13843018ac584914a8d4707" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:12.223094 4508 raft_consensus.cc:1261] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:12.223475 9105 consensus_queue.cc:1035] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:12.223515 7017 consensus_queue.cc:1035] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 20:46:12.223666 4572 consensus_peers.cc:485] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 76b6dee18d224e879bf0d3b6bbb2b9b5. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:12.224258 6703 raft_consensus.cc:2932] T 8602022a45e1482ebf8dab9930ce8209 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:12.224440 4374 raft_consensus.cc:1261] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:12.224853 9042 consensus_queue.cc:1035] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:12.226577 6815 tablet_service.cc:1514] Processing DeleteTablet for tablet 8602022a45e1482ebf8dab9930ce8209 with delete_type TABLET_DATA_TOMBSTONED (TS a4bf1df7fa6b4948be502c182bfbb833 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35290
I20240502 20:46:12.226825 9042 raft_consensus.cc:2932] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } } }
I20240502 20:46:12.227159 4374 raft_consensus.cc:2932] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } } }
I20240502 20:46:12.229104 10147 tablet_replica.cc:330] T 8602022a45e1482ebf8dab9930ce8209 P a4bf1df7fa6b4948be502c182bfbb833: stopping tablet replica
I20240502 20:46:12.229266 10147 raft_consensus.cc:2229] T 8602022a45e1482ebf8dab9930ce8209 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:12.229514 10147 raft_consensus.cc:2258] T 8602022a45e1482ebf8dab9930ce8209 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:12.229708 4228 catalog_manager.cc:5484] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index -1 to 4, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:12.233358 10147 ts_tablet_manager.cc:1875] T 8602022a45e1482ebf8dab9930ce8209 P a4bf1df7fa6b4948be502c182bfbb833: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:12.233342 4637 raft_consensus.cc:2932] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:12.236356 10147 ts_tablet_manager.cc:1888] T 8602022a45e1482ebf8dab9930ce8209 P a4bf1df7fa6b4948be502c182bfbb833: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 20:46:12.236479 10147 log.cc:1196] T 8602022a45e1482ebf8dab9930ce8209 P a4bf1df7fa6b4948be502c182bfbb833: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/wal/wals/8602022a45e1482ebf8dab9930ce8209
I20240502 20:46:12.237068 4220 catalog_manager.cc:4830] TS a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867): tablet 8602022a45e1482ebf8dab9930ce8209 (table rebalancer_extra_table_6 [id=f1553e3c7d3c4023865edad13ab86f4a]) successfully deleted
I20240502 20:46:12.238440 4228 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 20:46:12.242028 4508 raft_consensus.cc:2932] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } } }
I20240502 20:46:12.245980 9105 consensus_queue.cc:1035] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:12.252415 10094 tablet.cc:1722] T f7553e0da1b24ff484267c2e6feefc35 P 201d2b6485084e7dad2cbb722af94779: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 20:46:12.252775 10094 tablet.cc:1620] T f7553e0da1b24ff484267c2e6feefc35 P 201d2b6485084e7dad2cbb722af94779: MemRowSet was empty: no flush needed.
I20240502 20:46:12.253085 10107 tablet.cc:1722] T f7553e0da1b24ff484267c2e6feefc35 P 923cfab926e7443f8ffd7b21ad0a9d91: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 20:46:12.253127 10095 tablet.cc:1722] T f7553e0da1b24ff484267c2e6feefc35 P 7845c397230d49358ba2fad9db53f97c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 20:46:12.253377 10095 tablet.cc:1620] T f7553e0da1b24ff484267c2e6feefc35 P 7845c397230d49358ba2fad9db53f97c: MemRowSet was empty: no flush needed.
I20240502 20:46:12.253377 10107 tablet.cc:1620] T f7553e0da1b24ff484267c2e6feefc35 P 923cfab926e7443f8ffd7b21ad0a9d91: MemRowSet was empty: no flush needed.
I20240502 20:46:12.255820 8907 raft_consensus.cc:1050] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc: attempting to promote NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 to VOTER
I20240502 20:46:12.256271 8907 consensus_queue.cc:237] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } }
I20240502 20:46:12.257845 6703 raft_consensus.cc:1261] T a62a7ce526f14c25994bb9e0e3c3bc84 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:12.258744 4228 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: rebalancer_dynamic_table [id=a14a0e13a5d8440899bdd3f13abdea69] alter complete (version 2)
I20240502 20:46:12.258981 8146 consensus_queue.cc:1035] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:12.260090 6835 raft_consensus.cc:1261] T a62a7ce526f14c25994bb9e0e3c3bc84 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEARNER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:12.260167 4374 raft_consensus.cc:1261] T a62a7ce526f14c25994bb9e0e3c3bc84 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:12.260541 8146 consensus_queue.cc:1035] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:12.260687 9936 consensus_queue.cc:1035] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:12.268016 8146 raft_consensus.cc:2932] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:12.268491 9929 raft_consensus.cc:1050] T 52e302882311488ab1cac90ad03fe270 P a4bf1df7fa6b4948be502c182bfbb833: attempting to promote NON_VOTER c3d26eafd51547acb780d4aac5ebe7cc to VOTER
I20240502 20:46:12.268922 9929 consensus_queue.cc:237] T 52e302882311488ab1cac90ad03fe270 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } }
I20240502 20:46:12.270560 6967 raft_consensus.cc:1261] T 52e302882311488ab1cac90ad03fe270 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Refusing update from remote peer a4bf1df7fa6b4948be502c182bfbb833: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:12.270586 7098 raft_consensus.cc:1261] T 52e302882311488ab1cac90ad03fe270 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Refusing update from remote peer a4bf1df7fa6b4948be502c182bfbb833: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:12.271032 9402 consensus_queue.cc:1035] T 52e302882311488ab1cac90ad03fe270 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:46:12.271361 9402 consensus_queue.cc:1035] T 52e302882311488ab1cac90ad03fe270 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:46:12.272238 6835 raft_consensus.cc:2932] T a62a7ce526f14c25994bb9e0e3c3bc84 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:12.272792 4233 catalog_manager.cc:5484] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: config changed from index 7 to 8, a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:12.275354 4508 raft_consensus.cc:1261] T 52e302882311488ab1cac90ad03fe270 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEARNER]: Refusing update from remote peer a4bf1df7fa6b4948be502c182bfbb833: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:12.275357 6703 raft_consensus.cc:2932] T a62a7ce526f14c25994bb9e0e3c3bc84 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:12.275692 4507 consensus_queue.cc:237] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } }
I20240502 20:46:12.275832 9033 consensus_queue.cc:1035] T 52e302882311488ab1cac90ad03fe270 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:46:12.276959 6702 raft_consensus.cc:1261] T a62a7ce526f14c25994bb9e0e3c3bc84 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:46:12.278103 6375 consensus_queue.cc:1035] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:12.278273 6831 raft_consensus.cc:1261] T a62a7ce526f14c25994bb9e0e3c3bc84 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:46:12.279168 4768 consensus_queue.cc:237] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } }
I20240502 20:46:12.279326 6375 consensus_queue.cc:1035] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:12.281140 6835 raft_consensus.cc:1261] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:12.281769 9907 consensus_queue.cc:1035] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:46:12.282055 6375 raft_consensus.cc:2932] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:12.282230 6967 raft_consensus.cc:1261] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:12.282745 9907 consensus_queue.cc:1035] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:46:12.283857 6831 raft_consensus.cc:2932] T a62a7ce526f14c25994bb9e0e3c3bc84 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:12.284289 4221 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a62a7ce526f14c25994bb9e0e3c3bc84 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:12.285063 10160 raft_consensus.cc:2932] T 52e302882311488ab1cac90ad03fe270 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } } }
I20240502 20:46:12.285226 4507 raft_consensus.cc:2932] T 52e302882311488ab1cac90ad03fe270 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } } }
W20240502 20:46:12.285691 4705 consensus_peers.cc:485] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd -> Peer c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097): Couldn't send request to peer c3d26eafd51547acb780d4aac5ebe7cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 78d87e2d8c0b4bbd8dd6d7f49cdb4287. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:12.287292 6957 raft_consensus.cc:2932] T 52e302882311488ab1cac90ad03fe270 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } } }
I20240502 20:46:12.287496 8185 raft_consensus.cc:1050] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2: attempting to promote NON_VOTER 7845c397230d49358ba2fad9db53f97c to VOTER
I20240502 20:46:12.287911 8185 consensus_queue.cc:237] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } }
I20240502 20:46:12.287868 7098 raft_consensus.cc:2932] T 52e302882311488ab1cac90ad03fe270 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } } }
I20240502 20:46:12.289657 6957 raft_consensus.cc:1261] T 93b7f7de1f8c49389c3d8c1e03c67f26 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:12.290086 6138 consensus_queue.cc:1035] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:46:12.290426 6703 raft_consensus.cc:1261] T 93b7f7de1f8c49389c3d8c1e03c67f26 P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:12.290445 4233 catalog_manager.cc:5484] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: config changed from index 8 to 9, VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) evicted. New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:12.290827 6138 consensus_queue.cc:1035] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:46:12.291332 4231 catalog_manager.cc:5484] T 52e302882311488ab1cac90ad03fe270 P a47e5c56f13843018ac584914a8d4707 reported cstate change: config changed from index 2 to 3, c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a4bf1df7fa6b4948be502c182bfbb833" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } } }
I20240502 20:46:12.292858 6831 raft_consensus.cc:1261] T 93b7f7de1f8c49389c3d8c1e03c67f26 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:12.293442 10168 consensus_queue.cc:1035] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:46:12.294548 9907 raft_consensus.cc:2932] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } } }
I20240502 20:46:12.297236 4354 tablet_service.cc:1514] Processing DeleteTablet for tablet a62a7ce526f14c25994bb9e0e3c3bc84 with delete_type TABLET_DATA_TOMBSTONED (TS c92e6e46782849479c8a0c6d82276fb2 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:38306
I20240502 20:46:12.297497 9956 tablet_replica.cc:330] T a62a7ce526f14c25994bb9e0e3c3bc84 P c92e6e46782849479c8a0c6d82276fb2: stopping tablet replica
I20240502 20:46:12.297658 9956 raft_consensus.cc:2229] T a62a7ce526f14c25994bb9e0e3c3bc84 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:12.297848 9956 pending_rounds.cc:62] T a62a7ce526f14c25994bb9e0e3c3bc84 P c92e6e46782849479c8a0c6d82276fb2: Trying to abort 1 pending ops.
I20240502 20:46:12.297884 9956 pending_rounds.cc:68] T a62a7ce526f14c25994bb9e0e3c3bc84 P c92e6e46782849479c8a0c6d82276fb2: Aborting op as it isn't in flight: id { term: 1 index: 8 } timestamp: 7023340635160555520 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "a62a7ce526f14c25994bb9e0e3c3bc84" old_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: NON_VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: true } } } new_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } } }
I20240502 20:46:12.298004 9956 raft_consensus.cc:2894] T a62a7ce526f14c25994bb9e0e3c3bc84 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Aborting config change with OpId 1.8: Aborted: Op aborted
I20240502 20:46:12.298079 9956 raft_consensus.cc:2866] T a62a7ce526f14c25994bb9e0e3c3bc84 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 20:46:12.298139 9956 raft_consensus.cc:2258] T a62a7ce526f14c25994bb9e0e3c3bc84 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:12.298431 6702 raft_consensus.cc:2932] T a62a7ce526f14c25994bb9e0e3c3bc84 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:12.298594 6835 raft_consensus.cc:2932] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } } }
I20240502 20:46:12.298928 9956 ts_tablet_manager.cc:1875] T a62a7ce526f14c25994bb9e0e3c3bc84 P c92e6e46782849479c8a0c6d82276fb2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:12.299237 4233 catalog_manager.cc:5484] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd reported cstate change: config changed from index -1 to 2, NON_VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) added. New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:12.301213 9956 ts_tablet_manager.cc:1888] T a62a7ce526f14c25994bb9e0e3c3bc84 P c92e6e46782849479c8a0c6d82276fb2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 20:46:12.303706 6835 consensus_queue.cc:237] T 52e302882311488ab1cac90ad03fe270 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } }
I20240502 20:46:12.304165 9956 log.cc:1196] T a62a7ce526f14c25994bb9e0e3c3bc84 P c92e6e46782849479c8a0c6d82276fb2: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal/wals/a62a7ce526f14c25994bb9e0e3c3bc84
I20240502 20:46:12.306041 8185 raft_consensus.cc:2932] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:12.306274 4507 raft_consensus.cc:1261] T 52e302882311488ab1cac90ad03fe270 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer a4bf1df7fa6b4948be502c182bfbb833: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:12.306769 6703 raft_consensus.cc:2932] T 93b7f7de1f8c49389c3d8c1e03c67f26 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:12.306959 7098 raft_consensus.cc:1261] T 52e302882311488ab1cac90ad03fe270 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Refusing update from remote peer a4bf1df7fa6b4948be502c182bfbb833: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:12.307271 6967 raft_consensus.cc:2932] T 93b7f7de1f8c49389c3d8c1e03c67f26 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:12.307904 9929 consensus_queue.cc:1035] T 52e302882311488ab1cac90ad03fe270 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:12.308056 4222 catalog_manager.cc:4830] TS c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827): tablet a62a7ce526f14c25994bb9e0e3c3bc84 (table rebalance_test_table_17 [id=75dfa602f9634de99d6f55e4fdeea21c]) successfully deleted
I20240502 20:46:12.309172 6835 raft_consensus.cc:2932] T 93b7f7de1f8c49389c3d8c1e03c67f26 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:12.310734 9033 consensus_queue.cc:1035] T 52e302882311488ab1cac90ad03fe270 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:12.310644 9402 raft_consensus.cc:2932] T 52e302882311488ab1cac90ad03fe270 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } } }
I20240502 20:46:12.311189 6967 raft_consensus.cc:2932] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } } }
I20240502 20:46:12.311806 4507 raft_consensus.cc:2932] T 52e302882311488ab1cac90ad03fe270 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } } }
I20240502 20:46:12.312139 4220 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 52e302882311488ab1cac90ad03fe270 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:12.312994 7098 raft_consensus.cc:2932] T 52e302882311488ab1cac90ad03fe270 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } } }
I20240502 20:46:12.313588 4233 catalog_manager.cc:5484] T 93b7f7de1f8c49389c3d8c1e03c67f26 P a47e5c56f13843018ac584914a8d4707 reported cstate change: config changed from index 2 to 3, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c92e6e46782849479c8a0c6d82276fb2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:12.315502 4227 catalog_manager.cc:5484] T 52e302882311488ab1cac90ad03fe270 P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: config changed from index 3 to 4, VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) evicted. New cstate: current_term: 1 leader_uuid: "a4bf1df7fa6b4948be502c182bfbb833" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } } }
I20240502 20:46:12.317791 4374 consensus_queue.cc:237] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } }
I20240502 20:46:12.319555 6947 tablet_service.cc:1514] Processing DeleteTablet for tablet 52e302882311488ab1cac90ad03fe270 with delete_type TABLET_DATA_TOMBSTONED (TS a47e5c56f13843018ac584914a8d4707 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45156
I20240502 20:46:12.319639 6703 raft_consensus.cc:1261] T 93b7f7de1f8c49389c3d8c1e03c67f26 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:12.319770 9898 tablet_replica.cc:330] T 52e302882311488ab1cac90ad03fe270 P a47e5c56f13843018ac584914a8d4707: stopping tablet replica
I20240502 20:46:12.319895 9898 raft_consensus.cc:2229] T 52e302882311488ab1cac90ad03fe270 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:12.319895 6967 raft_consensus.cc:1261] T 93b7f7de1f8c49389c3d8c1e03c67f26 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:12.320060 9898 raft_consensus.cc:2258] T 52e302882311488ab1cac90ad03fe270 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:12.320169 10169 consensus_queue.cc:1035] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:12.320387 6138 consensus_queue.cc:1035] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:12.320690 9898 ts_tablet_manager.cc:1875] T 52e302882311488ab1cac90ad03fe270 P a47e5c56f13843018ac584914a8d4707: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:12.321748 10138 raft_consensus.cc:1050] T ef4002126e8847809add182f262d62d9 P a47e5c56f13843018ac584914a8d4707: attempting to promote NON_VOTER ed47c023452a497192f35d2f617bf9bd to VOTER
I20240502 20:46:12.322057 10138 consensus_queue.cc:237] T ef4002126e8847809add182f262d62d9 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } }
I20240502 20:46:12.323000 9898 ts_tablet_manager.cc:1888] T 52e302882311488ab1cac90ad03fe270 P a47e5c56f13843018ac584914a8d4707: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 20:46:12.323074 9898 log.cc:1196] T 52e302882311488ab1cac90ad03fe270 P a47e5c56f13843018ac584914a8d4707: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/wal/wals/52e302882311488ab1cac90ad03fe270
I20240502 20:46:12.323485 4374 raft_consensus.cc:1261] T ef4002126e8847809add182f262d62d9 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer a47e5c56f13843018ac584914a8d4707: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:12.323572 4222 catalog_manager.cc:4830] TS a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): tablet 52e302882311488ab1cac90ad03fe270 (table rebalancer_extra_table_8 [id=553d578c620843f4b11d547972b8d776]) successfully deleted
I20240502 20:46:12.323957 4768 raft_consensus.cc:1261] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd [term 1 LEARNER]: Refusing update from remote peer a47e5c56f13843018ac584914a8d4707: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:12.324433 10168 raft_consensus.cc:2932] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:12.324681 10138 consensus_queue.cc:1035] T ef4002126e8847809add182f262d62d9 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:46:12.324756 6703 raft_consensus.cc:2932] T 93b7f7de1f8c49389c3d8c1e03c67f26 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:12.325037 10138 consensus_queue.cc:1035] T ef4002126e8847809add182f262d62d9 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:46:12.327440 6967 raft_consensus.cc:2932] T 93b7f7de1f8c49389c3d8c1e03c67f26 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:12.327731 4230 catalog_manager.cc:5484] T 93b7f7de1f8c49389c3d8c1e03c67f26 P 7845c397230d49358ba2fad9db53f97c reported cstate change: config changed from index 3 to 4, VOTER a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) evicted. New cstate: current_term: 1 leader_uuid: "c92e6e46782849479c8a0c6d82276fb2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:12.328469 10138 raft_consensus.cc:2932] T ef4002126e8847809add182f262d62d9 P a47e5c56f13843018ac584914a8d4707 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ed47c023452a497192f35d2f617bf9bd (127.30.32.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } } }
I20240502 20:46:12.328881 4768 raft_consensus.cc:2932] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ed47c023452a497192f35d2f617bf9bd (127.30.32.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } } }
I20240502 20:46:12.330294 4374 raft_consensus.cc:2932] T ef4002126e8847809add182f262d62d9 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ed47c023452a497192f35d2f617bf9bd (127.30.32.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } } }
I20240502 20:46:12.331295 4231 catalog_manager.cc:5484] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd reported cstate change: config changed from index 2 to 3, ed47c023452a497192f35d2f617bf9bd (127.30.32.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a47e5c56f13843018ac584914a8d4707" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } } }
I20240502 20:46:12.331665 6815 tablet_service.cc:1514] Processing DeleteTablet for tablet 93b7f7de1f8c49389c3d8c1e03c67f26 with delete_type TABLET_DATA_TOMBSTONED (TS a4bf1df7fa6b4948be502c182bfbb833 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35290
I20240502 20:46:12.331912 10147 tablet_replica.cc:330] T 93b7f7de1f8c49389c3d8c1e03c67f26 P a4bf1df7fa6b4948be502c182bfbb833: stopping tablet replica
I20240502 20:46:12.332052 10147 raft_consensus.cc:2229] T 93b7f7de1f8c49389c3d8c1e03c67f26 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:12.332213 10147 raft_consensus.cc:2258] T 93b7f7de1f8c49389c3d8c1e03c67f26 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:12.333375 10147 ts_tablet_manager.cc:1875] T 93b7f7de1f8c49389c3d8c1e03c67f26 P a4bf1df7fa6b4948be502c182bfbb833: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:12.335139 10147 ts_tablet_manager.cc:1888] T 93b7f7de1f8c49389c3d8c1e03c67f26 P a4bf1df7fa6b4948be502c182bfbb833: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 20:46:12.335191 10147 log.cc:1196] T 93b7f7de1f8c49389c3d8c1e03c67f26 P a4bf1df7fa6b4948be502c182bfbb833: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/wal/wals/93b7f7de1f8c49389c3d8c1e03c67f26
I20240502 20:46:12.335647 4220 catalog_manager.cc:4830] TS a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867): tablet 93b7f7de1f8c49389c3d8c1e03c67f26 (table rebalancer_extra_table_8 [id=553d578c620843f4b11d547972b8d776]) successfully deleted
I20240502 20:46:12.337200 4222 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 93b7f7de1f8c49389c3d8c1e03c67f26 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:12.323989 6835 raft_consensus.cc:1261] T ef4002126e8847809add182f262d62d9 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Refusing update from remote peer a47e5c56f13843018ac584914a8d4707: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:12.337719 9156 consensus_queue.cc:1035] T ef4002126e8847809add182f262d62d9 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:46:12.338984 6835 raft_consensus.cc:2932] T ef4002126e8847809add182f262d62d9 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ed47c023452a497192f35d2f617bf9bd (127.30.32.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } } }
I20240502 20:46:12.361588 4231 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 20:46:12.367484 10094 tablet.cc:1722] T f7553e0da1b24ff484267c2e6feefc35 P 201d2b6485084e7dad2cbb722af94779: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 20:46:12.368180 10095 tablet.cc:1722] T f7553e0da1b24ff484267c2e6feefc35 P 7845c397230d49358ba2fad9db53f97c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 20:46:12.370175 10107 tablet.cc:1722] T f7553e0da1b24ff484267c2e6feefc35 P 923cfab926e7443f8ffd7b21ad0a9d91: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 20:46:12.371479 9907 raft_consensus.cc:1050] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd: attempting to promote NON_VOTER 201d2b6485084e7dad2cbb722af94779 to VOTER
I20240502 20:46:12.371984 9907 consensus_queue.cc:237] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } }
I20240502 20:46:12.374475 4231 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: rebalancer_dynamic_table_new_name [id=a14a0e13a5d8440899bdd3f13abdea69] alter complete (version 3)
I20240502 20:46:12.374926 4768 raft_consensus.cc:1203] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 20:46:12.376212 4374 raft_consensus.cc:1261] T 20373b53281a4214b27f014396f7a651 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:12.376610 4637 raft_consensus.cc:1261] T 20373b53281a4214b27f014396f7a651 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:12.376858 7098 raft_consensus.cc:1261] T 20373b53281a4214b27f014396f7a651 P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:12.377115 9908 consensus_queue.cc:1035] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:12.377491 9125 consensus_queue.cc:1035] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:12.378089 9908 consensus_queue.cc:1035] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:12.380651 9908 raft_consensus.cc:2932] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:12.381196 4374 raft_consensus.cc:2932] T 20373b53281a4214b27f014396f7a651 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:12.381537 7099 raft_consensus.cc:2932] T 20373b53281a4214b27f014396f7a651 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:12.383991 4230 catalog_manager.cc:5484] T 20373b53281a4214b27f014396f7a651 P c92e6e46782849479c8a0c6d82276fb2 reported cstate change: config changed from index 4 to 5, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:12.386925 4637 raft_consensus.cc:2932] T 20373b53281a4214b27f014396f7a651 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:12.391745 4768 consensus_queue.cc:237] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } }
I20240502 20:46:12.393344 7099 raft_consensus.cc:1261] T 20373b53281a4214b27f014396f7a651 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:12.393523 4377 raft_consensus.cc:1261] T 20373b53281a4214b27f014396f7a651 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:12.393827 9125 consensus_queue.cc:1035] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:12.394011 9907 consensus_queue.cc:1035] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:12.395782 9907 raft_consensus.cc:2932] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:12.396239 7099 raft_consensus.cc:2932] T 20373b53281a4214b27f014396f7a651 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:12.397244 4222 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 20373b53281a4214b27f014396f7a651 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:12.398614 4374 raft_consensus.cc:2932] T 20373b53281a4214b27f014396f7a651 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:12.398633 4228 catalog_manager.cc:5484] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd reported cstate change: config changed from index 5 to 6, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:12.400092 9907 raft_consensus.cc:1050] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd: attempting to promote NON_VOTER a47e5c56f13843018ac584914a8d4707 to VOTER
I20240502 20:46:12.400466 9907 consensus_queue.cc:237] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } }
I20240502 20:46:12.402388 6967 raft_consensus.cc:1261] T d37fab152a8a44868d3160ec3057bb98 P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:12.402489 4618 tablet_service.cc:1514] Processing DeleteTablet for tablet 20373b53281a4214b27f014396f7a651 with delete_type TABLET_DATA_TOMBSTONED (TS 923cfab926e7443f8ffd7b21ad0a9d91 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57068
I20240502 20:46:12.402614 4637 raft_consensus.cc:1261] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:12.402786 6703 raft_consensus.cc:1261] T d37fab152a8a44868d3160ec3057bb98 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:12.403900 10217 consensus_queue.cc:1035] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:12.404723 10225 tablet_replica.cc:330] T 20373b53281a4214b27f014396f7a651 P 923cfab926e7443f8ffd7b21ad0a9d91: stopping tablet replica
I20240502 20:46:12.404953 10225 raft_consensus.cc:2229] T 20373b53281a4214b27f014396f7a651 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:12.405690 10225 raft_consensus.cc:2258] T 20373b53281a4214b27f014396f7a651 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:12.406260 10209 consensus_queue.cc:1035] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:12.406405 10225 ts_tablet_manager.cc:1875] T 20373b53281a4214b27f014396f7a651 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:12.406698 10209 consensus_queue.cc:1035] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:12.409855 9907 raft_consensus.cc:2932] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:12.411785 10225 ts_tablet_manager.cc:1888] T 20373b53281a4214b27f014396f7a651 P 923cfab926e7443f8ffd7b21ad0a9d91: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:46:12.411863 10225 log.cc:1196] T 20373b53281a4214b27f014396f7a651 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal/wals/20373b53281a4214b27f014396f7a651
I20240502 20:46:12.412532 4219 catalog_manager.cc:4830] TS 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141): tablet 20373b53281a4214b27f014396f7a651 (table rebalance_test_table_3 [id=54d5208bcf1c43998e3ac81026ea9fbe]) successfully deleted
I20240502 20:46:12.412703 6703 raft_consensus.cc:2932] T d37fab152a8a44868d3160ec3057bb98 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:12.413163 4231 catalog_manager.cc:5484] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd reported cstate change: config changed from index 7 to 8, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:12.414539 6967 raft_consensus.cc:2932] T d37fab152a8a44868d3160ec3057bb98 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:12.418901 4637 raft_consensus.cc:2932] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:12.434680 4768 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d37fab152a8a44868d3160ec3057bb98"
dest_uuid: "ed47c023452a497192f35d2f617bf9bd"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56214
I20240502 20:46:12.434860 4768 raft_consensus.cc:595] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Received request to transfer leadership
I20240502 20:46:12.464757 6703 consensus_queue.cc:237] T 3d938d01d5154590a9262acd54094ad2 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } }
I20240502 20:46:12.466353 7099 raft_consensus.cc:1261] T 3d938d01d5154590a9262acd54094ad2 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Refusing update from remote peer 7845c397230d49358ba2fad9db53f97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:12.466418 6967 raft_consensus.cc:1261] T 3d938d01d5154590a9262acd54094ad2 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Refusing update from remote peer 7845c397230d49358ba2fad9db53f97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 20:46:12.466713 6636 consensus_peers.cc:485] T 3d938d01d5154590a9262acd54094ad2 P 7845c397230d49358ba2fad9db53f97c -> Peer 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141): Couldn't send request to peer 923cfab926e7443f8ffd7b21ad0a9d91. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3d938d01d5154590a9262acd54094ad2. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:12.466816 10270 consensus_queue.cc:1035] T 3d938d01d5154590a9262acd54094ad2 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:46:12.467000 9297 consensus_queue.cc:1035] T 3d938d01d5154590a9262acd54094ad2 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:46:12.469291 9297 raft_consensus.cc:2932] T 3d938d01d5154590a9262acd54094ad2 P 7845c397230d49358ba2fad9db53f97c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } } }
I20240502 20:46:12.469504 6835 raft_consensus.cc:1203] T e7e0539e60064d97a7eb0eacd9df065e P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:12.469754 6967 raft_consensus.cc:2932] T 3d938d01d5154590a9262acd54094ad2 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } } }
I20240502 20:46:12.471874 7099 raft_consensus.cc:2932] T 3d938d01d5154590a9262acd54094ad2 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } } }
I20240502 20:46:12.473148 7017 raft_consensus.cc:1050] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91: attempting to promote NON_VOTER 201d2b6485084e7dad2cbb722af94779 to VOTER
I20240502 20:46:12.473551 7017 consensus_queue.cc:237] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } }
I20240502 20:46:12.474745 4228 catalog_manager.cc:5484] T 3d938d01d5154590a9262acd54094ad2 P a47e5c56f13843018ac584914a8d4707 reported cstate change: config changed from index -1 to 2, NON_VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) added. New cstate: current_term: 1 leader_uuid: "7845c397230d49358ba2fad9db53f97c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } } }
I20240502 20:46:12.475346 4768 raft_consensus.cc:1261] T a7ea68e8581a4cd5a6212abc96dfbcde P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:12.475759 4374 raft_consensus.cc:1261] T a7ea68e8581a4cd5a6212abc96dfbcde P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:12.476377 9041 consensus_queue.cc:1035] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:12.476727 9041 consensus_queue.cc:1035] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:12.477311 7099 raft_consensus.cc:1261] T a7ea68e8581a4cd5a6212abc96dfbcde P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:12.477701 9043 consensus_queue.cc:1035] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:12.479197 4227 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 20:46:12.479351 4227 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 20:46:12.481720 4227 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: Sending DeleteTablet for 3 replicas of tablet f7553e0da1b24ff484267c2e6feefc35
I20240502 20:46:12.482407 4618 tablet_service.cc:1514] Processing DeleteTablet for tablet f7553e0da1b24ff484267c2e6feefc35 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:12 UTC) from {username='slave'} at 127.0.0.1:57068
I20240502 20:46:12.482474 6683 tablet_service.cc:1514] Processing DeleteTablet for tablet f7553e0da1b24ff484267c2e6feefc35 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:12 UTC) from {username='slave'} at 127.0.0.1:37506
I20240502 20:46:12.482681 7079 tablet_service.cc:1514] Processing DeleteTablet for tablet f7553e0da1b24ff484267c2e6feefc35 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:12 UTC) from {username='slave'} at 127.0.0.1:59162
I20240502 20:46:12.482766 10225 tablet_replica.cc:330] T f7553e0da1b24ff484267c2e6feefc35 P 923cfab926e7443f8ffd7b21ad0a9d91: stopping tablet replica
I20240502 20:46:12.482797 10284 tablet_replica.cc:330] T f7553e0da1b24ff484267c2e6feefc35 P 7845c397230d49358ba2fad9db53f97c: stopping tablet replica
I20240502 20:46:12.482915 10225 raft_consensus.cc:2229] T f7553e0da1b24ff484267c2e6feefc35 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:12.482933 10284 raft_consensus.cc:2229] T f7553e0da1b24ff484267c2e6feefc35 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:12.483031 10284 raft_consensus.cc:2258] T f7553e0da1b24ff484267c2e6feefc35 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:12.483034 10225 raft_consensus.cc:2258] T f7553e0da1b24ff484267c2e6feefc35 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:12.483687 10225 ts_tablet_manager.cc:1875] T f7553e0da1b24ff484267c2e6feefc35 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:12.483704 10284 ts_tablet_manager.cc:1875] T f7553e0da1b24ff484267c2e6feefc35 P 7845c397230d49358ba2fad9db53f97c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:12.483749 9041 raft_consensus.cc:2932] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:12.484740 7099 raft_consensus.cc:2932] T a7ea68e8581a4cd5a6212abc96dfbcde P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:12.485968 10284 ts_tablet_manager.cc:1888] T f7553e0da1b24ff484267c2e6feefc35 P 7845c397230d49358ba2fad9db53f97c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:12.486109 10284 log.cc:1196] T f7553e0da1b24ff484267c2e6feefc35 P 7845c397230d49358ba2fad9db53f97c: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/wal/wals/f7553e0da1b24ff484267c2e6feefc35
I20240502 20:46:12.486277 4374 raft_consensus.cc:2932] T a7ea68e8581a4cd5a6212abc96dfbcde P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:12.486639 10284 ts_tablet_manager.cc:1909] T f7553e0da1b24ff484267c2e6feefc35 P 7845c397230d49358ba2fad9db53f97c: Deleting consensus metadata
I20240502 20:46:12.487437 4221 catalog_manager.cc:4830] TS 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): tablet f7553e0da1b24ff484267c2e6feefc35 (table rebalancer_dynamic_table_new_name [id=a14a0e13a5d8440899bdd3f13abdea69]) successfully deleted
I20240502 20:46:12.488256 4227 catalog_manager.cc:5484] T a7ea68e8581a4cd5a6212abc96dfbcde P 201d2b6485084e7dad2cbb722af94779 reported cstate change: config changed from index 4 to 5, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:12.488953 10225 ts_tablet_manager.cc:1888] T f7553e0da1b24ff484267c2e6feefc35 P 923cfab926e7443f8ffd7b21ad0a9d91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:12.489151 10225 log.cc:1196] T f7553e0da1b24ff484267c2e6feefc35 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal/wals/f7553e0da1b24ff484267c2e6feefc35
I20240502 20:46:12.489459 10225 ts_tablet_manager.cc:1909] T f7553e0da1b24ff484267c2e6feefc35 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting consensus metadata
I20240502 20:46:12.489732 4768 raft_consensus.cc:2932] T a7ea68e8581a4cd5a6212abc96dfbcde P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:12.490684 4219 catalog_manager.cc:4830] TS 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141): tablet f7553e0da1b24ff484267c2e6feefc35 (table rebalancer_dynamic_table_new_name [id=a14a0e13a5d8440899bdd3f13abdea69]) successfully deleted
I20240502 20:46:12.492257 10285 tablet_replica.cc:330] T f7553e0da1b24ff484267c2e6feefc35 P 201d2b6485084e7dad2cbb722af94779: stopping tablet replica
I20240502 20:46:12.493181 10285 raft_consensus.cc:2229] T f7553e0da1b24ff484267c2e6feefc35 P 201d2b6485084e7dad2cbb722af94779 [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:12.493368 10285 raft_consensus.cc:2258] T f7553e0da1b24ff484267c2e6feefc35 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:12.493886 10285 ts_tablet_manager.cc:1875] T f7553e0da1b24ff484267c2e6feefc35 P 201d2b6485084e7dad2cbb722af94779: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:12.495999 10285 ts_tablet_manager.cc:1888] T f7553e0da1b24ff484267c2e6feefc35 P 201d2b6485084e7dad2cbb722af94779: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:12.496078 10285 log.cc:1196] T f7553e0da1b24ff484267c2e6feefc35 P 201d2b6485084e7dad2cbb722af94779: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/wal/wals/f7553e0da1b24ff484267c2e6feefc35
I20240502 20:46:12.496336 10285 ts_tablet_manager.cc:1909] T f7553e0da1b24ff484267c2e6feefc35 P 201d2b6485084e7dad2cbb722af94779: Deleting consensus metadata
I20240502 20:46:12.496963 4219 catalog_manager.cc:4830] TS 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): tablet f7553e0da1b24ff484267c2e6feefc35 (table rebalancer_dynamic_table_new_name [id=a14a0e13a5d8440899bdd3f13abdea69]) successfully deleted
I20240502 20:46:12.578609 4618 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 20:46:12.578639 6947 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:12.579313 6683 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:46:12.589159 4487 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 20:46:12.589241 4227 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41812:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 20:46:12.594274 4507 consensus_queue.cc:237] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } }
I20240502 20:46:12.595582 6967 raft_consensus.cc:1261] T 8565ebbd21dd4f708a0254d1ed2d6670 P a47e5c56f13843018ac584914a8d4707 [term 2 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 20:46:12.595636 6703 raft_consensus.cc:1261] T 8565ebbd21dd4f708a0254d1ed2d6670 P 7845c397230d49358ba2fad9db53f97c [term 2 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 20:46:12.596274 8146 consensus_queue.cc:1035] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:46:12.596654 6703 raft_consensus.cc:1203] T c9d25657e9a14b35b2f6e8d82f4533c7 P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:12.598407 9936 raft_consensus.cc:2932] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:12.598889 6967 raft_consensus.cc:2932] T 8565ebbd21dd4f708a0254d1ed2d6670 P a47e5c56f13843018ac584914a8d4707 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:12.600059 9936 consensus_queue.cc:1035] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:46:12.601491 4221 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8565ebbd21dd4f708a0254d1ed2d6670 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:12.602063 4750 tablet_service.cc:1467] Processing CreateTablet for tablet aa5a1d063b7144e1965e4157a7cefca3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=08c4df4d5a16481a9b470989a084c5df]), partition=
I20240502 20:46:12.602319 4750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa5a1d063b7144e1965e4157a7cefca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:12.604624 10326 tablet_bootstrap.cc:492] T aa5a1d063b7144e1965e4157a7cefca3 P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:46:12.605262 4618 tablet_service.cc:1467] Processing CreateTablet for tablet aa5a1d063b7144e1965e4157a7cefca3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=08c4df4d5a16481a9b470989a084c5df]), partition=
I20240502 20:46:12.605517 4618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa5a1d063b7144e1965e4157a7cefca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:12.606604 10326 tablet_bootstrap.cc:654] T aa5a1d063b7144e1965e4157a7cefca3 P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:12.609544 10328 tablet_bootstrap.cc:492] T aa5a1d063b7144e1965e4157a7cefca3 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:46:12.609781 7079 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:46:12.610543 6947 tablet_service.cc:1467] Processing CreateTablet for tablet aa5a1d063b7144e1965e4157a7cefca3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=08c4df4d5a16481a9b470989a084c5df]), partition=
I20240502 20:46:12.610811 6947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa5a1d063b7144e1965e4157a7cefca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:12.611083 10328 tablet_bootstrap.cc:654] T aa5a1d063b7144e1965e4157a7cefca3 P 923cfab926e7443f8ffd7b21ad0a9d91: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:12.612248 10328 tablet_bootstrap.cc:492] T aa5a1d063b7144e1965e4157a7cefca3 P 923cfab926e7443f8ffd7b21ad0a9d91: No bootstrap required, opened a new log
I20240502 20:46:12.612250 10326 tablet_bootstrap.cc:492] T aa5a1d063b7144e1965e4157a7cefca3 P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:46:12.612367 10328 ts_tablet_manager.cc:1367] T aa5a1d063b7144e1965e4157a7cefca3 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:46:12.612358 10326 ts_tablet_manager.cc:1367] T aa5a1d063b7144e1965e4157a7cefca3 P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 20:46:12.612872 10326 raft_consensus.cc:348] T aa5a1d063b7144e1965e4157a7cefca3 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:12.612871 10328 raft_consensus.cc:348] T aa5a1d063b7144e1965e4157a7cefca3 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:12.613018 10326 raft_consensus.cc:374] T aa5a1d063b7144e1965e4157a7cefca3 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:12.613063 10326 raft_consensus.cc:729] T aa5a1d063b7144e1965e4157a7cefca3 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:46:12.613214 10326 consensus_queue.cc:260] T aa5a1d063b7144e1965e4157a7cefca3 P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:12.613343 10328 raft_consensus.cc:374] T aa5a1d063b7144e1965e4157a7cefca3 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:12.613406 10328 raft_consensus.cc:729] T aa5a1d063b7144e1965e4157a7cefca3 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: FOLLOWER
I20240502 20:46:12.613548 10326 ts_tablet_manager.cc:1398] T aa5a1d063b7144e1965e4157a7cefca3 P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:12.613575 10328 consensus_queue.cc:260] T aa5a1d063b7144e1965e4157a7cefca3 P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:12.614672 10328 ts_tablet_manager.cc:1398] T aa5a1d063b7144e1965e4157a7cefca3 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 20:46:12.618729 4231 catalog_manager.cc:5484] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: config changed from index 8 to 10, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New cstate: current_term: 2 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:12.622382 6703 raft_consensus.cc:2932] T 8565ebbd21dd4f708a0254d1ed2d6670 P 7845c397230d49358ba2fad9db53f97c [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:12.622540 4618 tablet_service.cc:1514] Processing DeleteTablet for tablet 8565ebbd21dd4f708a0254d1ed2d6670 with delete_type TABLET_DATA_TOMBSTONED (TS 923cfab926e7443f8ffd7b21ad0a9d91 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:57068
I20240502 20:46:12.622996 10225 tablet_replica.cc:330] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91: stopping tablet replica
I20240502 20:46:12.623149 10225 raft_consensus.cc:2229] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:12.623253 10225 raft_consensus.cc:2258] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:12.625509 10225 ts_tablet_manager.cc:1875] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:12.627794 10225 ts_tablet_manager.cc:1888] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 20:46:12.627857 10225 log.cc:1196] T 8565ebbd21dd4f708a0254d1ed2d6670 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal/wals/8565ebbd21dd4f708a0254d1ed2d6670
I20240502 20:46:12.630892 4219 catalog_manager.cc:4830] TS 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141): tablet 8565ebbd21dd4f708a0254d1ed2d6670 (table rebalance_test_table_11 [id=105c28a926194705956796da40cf3dab]) successfully deleted
I20240502 20:46:12.634085 6815 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:46:12.635797 4354 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:12.640585 4750 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:46:12.642283 9155 consensus_queue.cc:1035] T a999f994b78a4b33a95ee7d5407cba20 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:12.649663 4228 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41812:
name: "rebalancer_extra_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:46:12.655030 10138 consensus_queue.cc:1035] T a999f994b78a4b33a95ee7d5407cba20 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:12.658089 10330 tablet_bootstrap.cc:492] T aa5a1d063b7144e1965e4157a7cefca3 P a47e5c56f13843018ac584914a8d4707: Bootstrap starting.
I20240502 20:46:12.658623 6815 tablet_service.cc:1467] Processing CreateTablet for tablet 2c67f965350a45f89511679d40f04cbb (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=f47287b2ed8647b8998f404171bb8d17]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:12.658905 6815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c67f965350a45f89511679d40f04cbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:12.659444 4356 tablet_service.cc:1467] Processing CreateTablet for tablet 3ba2f5d327f3403ea1604aae9485b960 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=f47287b2ed8647b8998f404171bb8d17]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:12.659533 4750 tablet_service.cc:1467] Processing CreateTablet for tablet 3ba2f5d327f3403ea1604aae9485b960 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=f47287b2ed8647b8998f404171bb8d17]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:12.659715 4356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ba2f5d327f3403ea1604aae9485b960. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:12.659818 4750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ba2f5d327f3403ea1604aae9485b960. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:12.662164 10326 tablet_bootstrap.cc:492] T 3ba2f5d327f3403ea1604aae9485b960 P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:46:12.663203 10326 tablet_bootstrap.cc:654] T 3ba2f5d327f3403ea1604aae9485b960 P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:12.664727 7079 tablet_service.cc:1467] Processing CreateTablet for tablet d78c02773e454ea0b79cf1282eda125b (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=f47287b2ed8647b8998f404171bb8d17]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:12.664774 7078 tablet_service.cc:1467] Processing CreateTablet for tablet 2c67f965350a45f89511679d40f04cbb (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=f47287b2ed8647b8998f404171bb8d17]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:12.665020 7079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d78c02773e454ea0b79cf1282eda125b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:12.665292 7078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c67f965350a45f89511679d40f04cbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:12.665241 6946 tablet_service.cc:1467] Processing CreateTablet for tablet 2c67f965350a45f89511679d40f04cbb (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=f47287b2ed8647b8998f404171bb8d17]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:12.665251 4354 tablet_service.cc:1467] Processing CreateTablet for tablet d78c02773e454ea0b79cf1282eda125b (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=f47287b2ed8647b8998f404171bb8d17]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:12.665506 4354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d78c02773e454ea0b79cf1282eda125b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:12.665508 6946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c67f965350a45f89511679d40f04cbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:12.668045 6945 tablet_service.cc:1467] Processing CreateTablet for tablet 3ba2f5d327f3403ea1604aae9485b960 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=f47287b2ed8647b8998f404171bb8d17]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:12.668272 6945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ba2f5d327f3403ea1604aae9485b960. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:12.670382 6947 tablet_service.cc:1467] Processing CreateTablet for tablet d78c02773e454ea0b79cf1282eda125b (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=f47287b2ed8647b8998f404171bb8d17]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:12.670625 6947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d78c02773e454ea0b79cf1282eda125b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:12.672626 10330 tablet_bootstrap.cc:654] T aa5a1d063b7144e1965e4157a7cefca3 P a47e5c56f13843018ac584914a8d4707: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:12.683552 10326 tablet_bootstrap.cc:492] T 3ba2f5d327f3403ea1604aae9485b960 P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:46:12.683667 10326 ts_tablet_manager.cc:1367] T 3ba2f5d327f3403ea1604aae9485b960 P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.001s
I20240502 20:46:12.684160 10326 raft_consensus.cc:348] T 3ba2f5d327f3403ea1604aae9485b960 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:12.684265 10326 raft_consensus.cc:374] T 3ba2f5d327f3403ea1604aae9485b960 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:12.684298 10326 raft_consensus.cc:729] T 3ba2f5d327f3403ea1604aae9485b960 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:46:12.684441 10326 consensus_queue.cc:260] T 3ba2f5d327f3403ea1604aae9485b960 P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:12.684751 10326 ts_tablet_manager.cc:1398] T 3ba2f5d327f3403ea1604aae9485b960 P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240502 20:46:12.685350 10338 tablet_bootstrap.cc:492] T d78c02773e454ea0b79cf1282eda125b P 201d2b6485084e7dad2cbb722af94779: Bootstrap starting.
I20240502 20:46:12.686614 10338 tablet_bootstrap.cc:654] T d78c02773e454ea0b79cf1282eda125b P 201d2b6485084e7dad2cbb722af94779: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:12.687767 10335 tablet_bootstrap.cc:492] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap starting.
I20240502 20:46:12.688789 10335 tablet_bootstrap.cc:654] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:12.688800 10338 tablet_bootstrap.cc:492] T d78c02773e454ea0b79cf1282eda125b P 201d2b6485084e7dad2cbb722af94779: No bootstrap required, opened a new log
I20240502 20:46:12.688891 10338 ts_tablet_manager.cc:1367] T d78c02773e454ea0b79cf1282eda125b P 201d2b6485084e7dad2cbb722af94779: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 20:46:12.689628 10336 tablet_bootstrap.cc:492] T 3ba2f5d327f3403ea1604aae9485b960 P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:46:12.690541 10336 tablet_bootstrap.cc:654] T 3ba2f5d327f3403ea1604aae9485b960 P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:12.689493 10338 raft_consensus.cc:348] T d78c02773e454ea0b79cf1282eda125b P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:12.690794 10338 raft_consensus.cc:374] T d78c02773e454ea0b79cf1282eda125b P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:12.690855 10338 raft_consensus.cc:729] T d78c02773e454ea0b79cf1282eda125b P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Initialized, Role: FOLLOWER
I20240502 20:46:12.691053 10338 consensus_queue.cc:260] T d78c02773e454ea0b79cf1282eda125b P 201d2b6485084e7dad2cbb722af94779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:12.692029 10336 tablet_bootstrap.cc:492] T 3ba2f5d327f3403ea1604aae9485b960 P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:46:12.692169 10336 ts_tablet_manager.cc:1367] T 3ba2f5d327f3403ea1604aae9485b960 P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 20:46:12.692255 10338 ts_tablet_manager.cc:1398] T d78c02773e454ea0b79cf1282eda125b P 201d2b6485084e7dad2cbb722af94779: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 20:46:12.692720 10336 raft_consensus.cc:348] T 3ba2f5d327f3403ea1604aae9485b960 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:12.693125 10336 raft_consensus.cc:374] T 3ba2f5d327f3403ea1604aae9485b960 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:12.693166 10332 ts_tablet_manager.cc:902] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 201d2b6485084e7dad2cbb722af94779: Initiating tablet copy from peer 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:46:12.693259 10336 raft_consensus.cc:729] T 3ba2f5d327f3403ea1604aae9485b960 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:46:12.693537 10336 consensus_queue.cc:260] T 3ba2f5d327f3403ea1604aae9485b960 P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:12.694504 10330 tablet_bootstrap.cc:492] T aa5a1d063b7144e1965e4157a7cefca3 P a47e5c56f13843018ac584914a8d4707: No bootstrap required, opened a new log
I20240502 20:46:12.694626 10330 ts_tablet_manager.cc:1367] T aa5a1d063b7144e1965e4157a7cefca3 P a47e5c56f13843018ac584914a8d4707: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.000s
I20240502 20:46:12.694765 10332 tablet_copy_client.cc:308] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 201d2b6485084e7dad2cbb722af94779: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.67:42141
I20240502 20:46:12.695086 10330 raft_consensus.cc:348] T aa5a1d063b7144e1965e4157a7cefca3 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:12.695201 10330 raft_consensus.cc:374] T aa5a1d063b7144e1965e4157a7cefca3 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:12.695233 10330 raft_consensus.cc:729] T aa5a1d063b7144e1965e4157a7cefca3 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Initialized, Role: FOLLOWER
I20240502 20:46:12.695367 10330 consensus_queue.cc:260] T aa5a1d063b7144e1965e4157a7cefca3 P a47e5c56f13843018ac584914a8d4707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:12.695672 10330 ts_tablet_manager.cc:1398] T aa5a1d063b7144e1965e4157a7cefca3 P a47e5c56f13843018ac584914a8d4707: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:12.696003 4658 tablet_copy_service.cc:140] P 923cfab926e7443f8ffd7b21ad0a9d91: Received BeginTabletCopySession request for tablet 76b6dee18d224e879bf0d3b6bbb2b9b5 from peer 201d2b6485084e7dad2cbb722af94779 ({username='slave'} at 127.30.32.72:36609)
I20240502 20:46:12.696100 4658 tablet_copy_service.cc:161] P 923cfab926e7443f8ffd7b21ad0a9d91: Beginning new tablet copy session on tablet 76b6dee18d224e879bf0d3b6bbb2b9b5 from peer 201d2b6485084e7dad2cbb722af94779 at {username='slave'} at 127.30.32.72:36609: session id = 201d2b6485084e7dad2cbb722af94779-76b6dee18d224e879bf0d3b6bbb2b9b5
I20240502 20:46:12.696503 10330 tablet_bootstrap.cc:492] T 2c67f965350a45f89511679d40f04cbb P a47e5c56f13843018ac584914a8d4707: Bootstrap starting.
I20240502 20:46:12.697082 4658 tablet_copy_source_session.cc:205] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:12.698446 10330 tablet_bootstrap.cc:654] T 2c67f965350a45f89511679d40f04cbb P a47e5c56f13843018ac584914a8d4707: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:12.699190 10332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76b6dee18d224e879bf0d3b6bbb2b9b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:12.699901 10338 tablet_bootstrap.cc:492] T 2c67f965350a45f89511679d40f04cbb P 201d2b6485084e7dad2cbb722af94779: Bootstrap starting.
I20240502 20:46:12.701153 10338 tablet_bootstrap.cc:654] T 2c67f965350a45f89511679d40f04cbb P 201d2b6485084e7dad2cbb722af94779: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:12.702289 10332 tablet_copy_client.cc:783] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 201d2b6485084e7dad2cbb722af94779: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:12.702476 10336 ts_tablet_manager.cc:1398] T 3ba2f5d327f3403ea1604aae9485b960 P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20240502 20:46:12.703331 10336 tablet_bootstrap.cc:492] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:46:12.703779 10335 tablet_bootstrap.cc:492] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833: No bootstrap required, opened a new log
I20240502 20:46:12.703888 10335 ts_tablet_manager.cc:1367] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240502 20:46:12.704355 10335 raft_consensus.cc:348] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:12.704550 10335 raft_consensus.cc:374] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:12.704653 10335 raft_consensus.cc:729] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Initialized, Role: FOLLOWER
I20240502 20:46:12.704872 10335 consensus_queue.cc:260] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:12.705618 10332 tablet_copy_client.cc:647] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 201d2b6485084e7dad2cbb722af94779: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:12.705857 10335 ts_tablet_manager.cc:1398] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:12.706085 10336 tablet_bootstrap.cc:654] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:12.706192 10348 ts_tablet_manager.cc:902] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P c3d26eafd51547acb780d4aac5ebe7cc: Initiating tablet copy from peer ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:46:12.706771 10348 tablet_copy_client.cc:308] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P c3d26eafd51547acb780d4aac5ebe7cc: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.68:38193
I20240502 20:46:12.707307 4790 tablet_copy_service.cc:140] P ed47c023452a497192f35d2f617bf9bd: Received BeginTabletCopySession request for tablet 78d87e2d8c0b4bbd8dd6d7f49cdb4287 from peer c3d26eafd51547acb780d4aac5ebe7cc ({username='slave'} at 127.30.32.66:41629)
I20240502 20:46:12.707412 4790 tablet_copy_service.cc:161] P ed47c023452a497192f35d2f617bf9bd: Beginning new tablet copy session on tablet 78d87e2d8c0b4bbd8dd6d7f49cdb4287 from peer c3d26eafd51547acb780d4aac5ebe7cc at {username='slave'} at 127.30.32.66:41629: session id = c3d26eafd51547acb780d4aac5ebe7cc-78d87e2d8c0b4bbd8dd6d7f49cdb4287
I20240502 20:46:12.708078 10332 tablet_copy_client.cc:523] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 201d2b6485084e7dad2cbb722af94779: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:12.708348 4790 tablet_copy_source_session.cc:205] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:12.709280 10330 tablet_bootstrap.cc:492] T 2c67f965350a45f89511679d40f04cbb P a47e5c56f13843018ac584914a8d4707: No bootstrap required, opened a new log
I20240502 20:46:12.709393 10330 ts_tablet_manager.cc:1367] T 2c67f965350a45f89511679d40f04cbb P a47e5c56f13843018ac584914a8d4707: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.002s
I20240502 20:46:12.709921 10330 raft_consensus.cc:348] T 2c67f965350a45f89511679d40f04cbb P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:12.710034 10330 raft_consensus.cc:374] T 2c67f965350a45f89511679d40f04cbb P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:12.710074 10330 raft_consensus.cc:729] T 2c67f965350a45f89511679d40f04cbb P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Initialized, Role: FOLLOWER
I20240502 20:46:12.710206 10330 consensus_queue.cc:260] T 2c67f965350a45f89511679d40f04cbb P a47e5c56f13843018ac584914a8d4707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:12.710467 10330 ts_tablet_manager.cc:1398] T 2c67f965350a45f89511679d40f04cbb P a47e5c56f13843018ac584914a8d4707: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:12.710604 10330 tablet_bootstrap.cc:492] T 3ba2f5d327f3403ea1604aae9485b960 P a47e5c56f13843018ac584914a8d4707: Bootstrap starting.
I20240502 20:46:12.711349 10330 tablet_bootstrap.cc:654] T 3ba2f5d327f3403ea1604aae9485b960 P a47e5c56f13843018ac584914a8d4707: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:12.713021 10330 tablet_bootstrap.cc:492] T 3ba2f5d327f3403ea1604aae9485b960 P a47e5c56f13843018ac584914a8d4707: No bootstrap required, opened a new log
I20240502 20:46:12.713179 10330 ts_tablet_manager.cc:1367] T 3ba2f5d327f3403ea1604aae9485b960 P a47e5c56f13843018ac584914a8d4707: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 20:46:12.713713 10330 raft_consensus.cc:348] T 3ba2f5d327f3403ea1604aae9485b960 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:12.713891 10330 raft_consensus.cc:374] T 3ba2f5d327f3403ea1604aae9485b960 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:12.713981 10330 raft_consensus.cc:729] T 3ba2f5d327f3403ea1604aae9485b960 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Initialized, Role: FOLLOWER
I20240502 20:46:12.714322 10330 consensus_queue.cc:260] T 3ba2f5d327f3403ea1604aae9485b960 P a47e5c56f13843018ac584914a8d4707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:12.715240 10330 ts_tablet_manager.cc:1398] T 3ba2f5d327f3403ea1604aae9485b960 P a47e5c56f13843018ac584914a8d4707: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:12.715447 10330 tablet_bootstrap.cc:492] T d78c02773e454ea0b79cf1282eda125b P a47e5c56f13843018ac584914a8d4707: Bootstrap starting.
I20240502 20:46:12.716533 10330 tablet_bootstrap.cc:654] T d78c02773e454ea0b79cf1282eda125b P a47e5c56f13843018ac584914a8d4707: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:12.718242 10348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78d87e2d8c0b4bbd8dd6d7f49cdb4287. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:12.719417 10332 tablet_bootstrap.cc:492] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 201d2b6485084e7dad2cbb722af94779: Bootstrap starting.
I20240502 20:46:12.720484 10348 tablet_copy_client.cc:783] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P c3d26eafd51547acb780d4aac5ebe7cc: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:12.720651 10348 tablet_copy_client.cc:647] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P c3d26eafd51547acb780d4aac5ebe7cc: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:12.722079 10348 tablet_copy_client.cc:523] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P c3d26eafd51547acb780d4aac5ebe7cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:12.722654 10330 tablet_bootstrap.cc:492] T d78c02773e454ea0b79cf1282eda125b P a47e5c56f13843018ac584914a8d4707: No bootstrap required, opened a new log
I20240502 20:46:12.722803 10330 ts_tablet_manager.cc:1367] T d78c02773e454ea0b79cf1282eda125b P a47e5c56f13843018ac584914a8d4707: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 20:46:12.723479 10348 tablet_bootstrap.cc:492] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:46:12.723394 10330 raft_consensus.cc:348] T d78c02773e454ea0b79cf1282eda125b P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:12.723618 10330 raft_consensus.cc:374] T d78c02773e454ea0b79cf1282eda125b P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:12.723698 10330 raft_consensus.cc:729] T d78c02773e454ea0b79cf1282eda125b P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Initialized, Role: FOLLOWER
I20240502 20:46:12.723878 10330 consensus_queue.cc:260] T d78c02773e454ea0b79cf1282eda125b P a47e5c56f13843018ac584914a8d4707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:12.725194 10330 ts_tablet_manager.cc:1398] T d78c02773e454ea0b79cf1282eda125b P a47e5c56f13843018ac584914a8d4707: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:12.726119 10338 tablet_bootstrap.cc:492] T 2c67f965350a45f89511679d40f04cbb P 201d2b6485084e7dad2cbb722af94779: No bootstrap required, opened a new log
I20240502 20:46:12.726219 10338 ts_tablet_manager.cc:1367] T 2c67f965350a45f89511679d40f04cbb P 201d2b6485084e7dad2cbb722af94779: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.000s
I20240502 20:46:12.726665 10338 raft_consensus.cc:348] T 2c67f965350a45f89511679d40f04cbb P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:12.726749 10338 raft_consensus.cc:374] T 2c67f965350a45f89511679d40f04cbb P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:12.726768 10338 raft_consensus.cc:729] T 2c67f965350a45f89511679d40f04cbb P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Initialized, Role: FOLLOWER
I20240502 20:46:12.726876 10338 consensus_queue.cc:260] T 2c67f965350a45f89511679d40f04cbb P 201d2b6485084e7dad2cbb722af94779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:12.727210 10338 ts_tablet_manager.cc:1398] T 2c67f965350a45f89511679d40f04cbb P 201d2b6485084e7dad2cbb722af94779: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:12.730692 10336 tablet_bootstrap.cc:492] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:46:12.730854 10336 ts_tablet_manager.cc:1367] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.000s
I20240502 20:46:12.731515 10336 raft_consensus.cc:348] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:12.731622 10336 raft_consensus.cc:374] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:12.731648 10336 raft_consensus.cc:729] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:46:12.731761 10336 consensus_queue.cc:260] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:12.731915 10348 tablet_bootstrap.cc:492] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:12.732174 10348 tablet_bootstrap.cc:492] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap complete.
I20240502 20:46:12.732322 10348 ts_tablet_manager.cc:1367] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20240502 20:46:12.732746 10336 ts_tablet_manager.cc:1398] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:12.732771 10348 raft_consensus.cc:348] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } }
I20240502 20:46:12.732890 10348 raft_consensus.cc:729] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: LEARNER
I20240502 20:46:12.733026 10348 consensus_queue.cc:260] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } }
I20240502 20:46:12.733331 10348 ts_tablet_manager.cc:1398] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:12.734107 4790 tablet_copy_service.cc:342] P ed47c023452a497192f35d2f617bf9bd: Request end of tablet copy session c3d26eafd51547acb780d4aac5ebe7cc-78d87e2d8c0b4bbd8dd6d7f49cdb4287 received from {username='slave'} at 127.30.32.66:41629
I20240502 20:46:12.734195 4790 tablet_copy_service.cc:433] P ed47c023452a497192f35d2f617bf9bd: ending tablet copy session c3d26eafd51547acb780d4aac5ebe7cc-78d87e2d8c0b4bbd8dd6d7f49cdb4287 on tablet 78d87e2d8c0b4bbd8dd6d7f49cdb4287 with peer c3d26eafd51547acb780d4aac5ebe7cc
I20240502 20:46:12.743674 8460 raft_consensus.cc:482] T 2c67f965350a45f89511679d40f04cbb P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:12.743774 8460 raft_consensus.cc:504] T 2c67f965350a45f89511679d40f04cbb P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:12.744154 8460 leader_election.cc:288] T 2c67f965350a45f89511679d40f04cbb P 201d2b6485084e7dad2cbb722af94779 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867)
I20240502 20:46:12.744493 6967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c67f965350a45f89511679d40f04cbb" candidate_uuid: "201d2b6485084e7dad2cbb722af94779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47e5c56f13843018ac584914a8d4707" is_pre_election: true
I20240502 20:46:12.744635 6967 raft_consensus.cc:2454] T 2c67f965350a45f89511679d40f04cbb P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 201d2b6485084e7dad2cbb722af94779 in term 0.
I20240502 20:46:12.744608 6831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c67f965350a45f89511679d40f04cbb" candidate_uuid: "201d2b6485084e7dad2cbb722af94779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833" is_pre_election: true
I20240502 20:46:12.744733 6831 raft_consensus.cc:2454] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 201d2b6485084e7dad2cbb722af94779 in term 0.
I20240502 20:46:12.744935 7035 leader_election.cc:302] T 2c67f965350a45f89511679d40f04cbb P 201d2b6485084e7dad2cbb722af94779 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201d2b6485084e7dad2cbb722af94779, a47e5c56f13843018ac584914a8d4707; no voters:
I20240502 20:46:12.745260 8460 raft_consensus.cc:2781] T 2c67f965350a45f89511679d40f04cbb P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:12.745322 8460 raft_consensus.cc:482] T 2c67f965350a45f89511679d40f04cbb P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:12.745363 8460 raft_consensus.cc:3037] T 2c67f965350a45f89511679d40f04cbb P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:12.746332 8460 raft_consensus.cc:504] T 2c67f965350a45f89511679d40f04cbb P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:12.746678 8460 leader_election.cc:288] T 2c67f965350a45f89511679d40f04cbb P 201d2b6485084e7dad2cbb722af94779 [CANDIDATE]: Term 1 election: Requested vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867)
I20240502 20:46:12.746963 6967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c67f965350a45f89511679d40f04cbb" candidate_uuid: "201d2b6485084e7dad2cbb722af94779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47e5c56f13843018ac584914a8d4707"
I20240502 20:46:12.747082 6967 raft_consensus.cc:3037] T 2c67f965350a45f89511679d40f04cbb P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:12.747202 6831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c67f965350a45f89511679d40f04cbb" candidate_uuid: "201d2b6485084e7dad2cbb722af94779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833"
I20240502 20:46:12.747323 6831 raft_consensus.cc:3037] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:12.748083 6967 raft_consensus.cc:2454] T 2c67f965350a45f89511679d40f04cbb P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 201d2b6485084e7dad2cbb722af94779 in term 1.
I20240502 20:46:12.748296 6831 raft_consensus.cc:2454] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 201d2b6485084e7dad2cbb722af94779 in term 1.
I20240502 20:46:12.748353 7035 leader_election.cc:302] T 2c67f965350a45f89511679d40f04cbb P 201d2b6485084e7dad2cbb722af94779 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201d2b6485084e7dad2cbb722af94779, a47e5c56f13843018ac584914a8d4707; no voters:
I20240502 20:46:12.748831 8460 raft_consensus.cc:2781] T 2c67f965350a45f89511679d40f04cbb P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:12.748945 8460 raft_consensus.cc:686] T 2c67f965350a45f89511679d40f04cbb P 201d2b6485084e7dad2cbb722af94779 [term 1 LEADER]: Becoming Leader. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Running, Role: LEADER
I20240502 20:46:12.749141 8460 consensus_queue.cc:237] T 2c67f965350a45f89511679d40f04cbb P 201d2b6485084e7dad2cbb722af94779 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:12.751736 4235 catalog_manager.cc:5484] T 2c67f965350a45f89511679d40f04cbb P 201d2b6485084e7dad2cbb722af94779 reported cstate change: term changed from 0 to 1, leader changed from <none> to 201d2b6485084e7dad2cbb722af94779 (127.30.32.72). New cstate: current_term: 1 leader_uuid: "201d2b6485084e7dad2cbb722af94779" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:12.755928 9908 raft_consensus.cc:482] T aa5a1d063b7144e1965e4157a7cefca3 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:12.756031 9908 raft_consensus.cc:504] T aa5a1d063b7144e1965e4157a7cefca3 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:12.756469 9908 leader_election.cc:288] T aa5a1d063b7144e1965e4157a7cefca3 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:46:12.756865 4637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5a1d063b7144e1965e4157a7cefca3" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" is_pre_election: true
I20240502 20:46:12.756932 6967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5a1d063b7144e1965e4157a7cefca3" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47e5c56f13843018ac584914a8d4707" is_pre_election: true
I20240502 20:46:12.757047 4637 raft_consensus.cc:2454] T aa5a1d063b7144e1965e4157a7cefca3 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 0.
I20240502 20:46:12.757131 6967 raft_consensus.cc:2454] T aa5a1d063b7144e1965e4157a7cefca3 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 0.
I20240502 20:46:12.757397 4703 leader_election.cc:302] T aa5a1d063b7144e1965e4157a7cefca3 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:46:12.757640 9908 raft_consensus.cc:2781] T aa5a1d063b7144e1965e4157a7cefca3 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:12.757707 9908 raft_consensus.cc:482] T aa5a1d063b7144e1965e4157a7cefca3 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:12.757746 9908 raft_consensus.cc:3037] T aa5a1d063b7144e1965e4157a7cefca3 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:12.758720 6703 consensus_queue.cc:237] T ac57ad308b174217ac74c4cb5e1c7285 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true } }
I20240502 20:46:12.758764 9908 raft_consensus.cc:504] T aa5a1d063b7144e1965e4157a7cefca3 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:12.759151 9908 leader_election.cc:288] T aa5a1d063b7144e1965e4157a7cefca3 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 election: Requested vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:46:12.759775 4637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5a1d063b7144e1965e4157a7cefca3" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91"
I20240502 20:46:12.759903 4637 raft_consensus.cc:3037] T aa5a1d063b7144e1965e4157a7cefca3 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:12.760102 6967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5a1d063b7144e1965e4157a7cefca3" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47e5c56f13843018ac584914a8d4707"
I20240502 20:46:12.760228 6967 raft_consensus.cc:3037] T aa5a1d063b7144e1965e4157a7cefca3 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:12.760612 4507 raft_consensus.cc:1261] T ac57ad308b174217ac74c4cb5e1c7285 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer 7845c397230d49358ba2fad9db53f97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 20:46:12.761106 6639 consensus_peers.cc:485] T ac57ad308b174217ac74c4cb5e1c7285 P 7845c397230d49358ba2fad9db53f97c -> Peer c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827): Couldn't send request to peer c92e6e46782849479c8a0c6d82276fb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ac57ad308b174217ac74c4cb5e1c7285. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:12.761281 6967 raft_consensus.cc:2454] T aa5a1d063b7144e1965e4157a7cefca3 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 1.
I20240502 20:46:12.761348 4637 raft_consensus.cc:2454] T aa5a1d063b7144e1965e4157a7cefca3 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 1.
I20240502 20:46:12.761619 7099 raft_consensus.cc:1261] T ac57ad308b174217ac74c4cb5e1c7285 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Refusing update from remote peer 7845c397230d49358ba2fad9db53f97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:12.761641 4703 leader_election.cc:302] T aa5a1d063b7144e1965e4157a7cefca3 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:46:12.761668 10270 consensus_queue.cc:1035] T ac57ad308b174217ac74c4cb5e1c7285 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:46:12.761817 9908 raft_consensus.cc:2781] T aa5a1d063b7144e1965e4157a7cefca3 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:12.761896 9908 raft_consensus.cc:686] T aa5a1d063b7144e1965e4157a7cefca3 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Becoming Leader. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Running, Role: LEADER
I20240502 20:46:12.762020 9908 consensus_queue.cc:237] T aa5a1d063b7144e1965e4157a7cefca3 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:12.763191 10270 consensus_queue.cc:1035] T ac57ad308b174217ac74c4cb5e1c7285 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:46:12.764137 9105 raft_consensus.cc:2932] T ac57ad308b174217ac74c4cb5e1c7285 P 7845c397230d49358ba2fad9db53f97c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true } } }
I20240502 20:46:12.765472 4235 catalog_manager.cc:5484] T aa5a1d063b7144e1965e4157a7cefca3 P ed47c023452a497192f35d2f617bf9bd reported cstate change: term changed from 0 to 1, leader changed from <none> to ed47c023452a497192f35d2f617bf9bd (127.30.32.68). New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:12.767254 7099 raft_consensus.cc:2932] T ac57ad308b174217ac74c4cb5e1c7285 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true } } }
I20240502 20:46:12.768159 4507 raft_consensus.cc:2932] T ac57ad308b174217ac74c4cb5e1c7285 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true } } }
I20240502 20:46:12.768283 4228 catalog_manager.cc:5484] T ac57ad308b174217ac74c4cb5e1c7285 P 7845c397230d49358ba2fad9db53f97c reported cstate change: config changed from index -1 to 2, NON_VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) added. New cstate: current_term: 1 leader_uuid: "7845c397230d49358ba2fad9db53f97c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:12.769483 10332 tablet_bootstrap.cc:492] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 201d2b6485084e7dad2cbb722af94779: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:12.769768 10332 tablet_bootstrap.cc:492] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 201d2b6485084e7dad2cbb722af94779: Bootstrap complete.
I20240502 20:46:12.769953 10332 ts_tablet_manager.cc:1367] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 201d2b6485084e7dad2cbb722af94779: Time spent bootstrapping tablet: real 0.051s user 0.027s sys 0.000s
I20240502 20:46:12.772173 10332 raft_consensus.cc:348] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } }
I20240502 20:46:12.772691 10332 raft_consensus.cc:729] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Initialized, Role: LEARNER
I20240502 20:46:12.772895 10332 consensus_queue.cc:260] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 201d2b6485084e7dad2cbb722af94779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } }
I20240502 20:46:12.773546 10332 ts_tablet_manager.cc:1398] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 201d2b6485084e7dad2cbb722af94779: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 20:46:12.774211 4658 tablet_copy_service.cc:342] P 923cfab926e7443f8ffd7b21ad0a9d91: Request end of tablet copy session 201d2b6485084e7dad2cbb722af94779-76b6dee18d224e879bf0d3b6bbb2b9b5 received from {username='slave'} at 127.30.32.72:36609
I20240502 20:46:12.774333 4658 tablet_copy_service.cc:433] P 923cfab926e7443f8ffd7b21ad0a9d91: ending tablet copy session 201d2b6485084e7dad2cbb722af94779-76b6dee18d224e879bf0d3b6bbb2b9b5 on tablet 76b6dee18d224e879bf0d3b6bbb2b9b5 with peer 201d2b6485084e7dad2cbb722af94779
I20240502 20:46:12.780670 6703 consensus_queue.cc:237] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: true } }
I20240502 20:46:12.782142 6967 raft_consensus.cc:1261] T a9afb75bfc114f59850eaea084a306f0 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Refusing update from remote peer 7845c397230d49358ba2fad9db53f97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:12.782442 6831 raft_consensus.cc:1261] T a9afb75bfc114f59850eaea084a306f0 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Refusing update from remote peer 7845c397230d49358ba2fad9db53f97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 20:46:12.782644 6639 consensus_peers.cc:485] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c -> Peer ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193): Couldn't send request to peer ed47c023452a497192f35d2f617bf9bd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a9afb75bfc114f59850eaea084a306f0. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:12.782902 9105 consensus_queue.cc:1035] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:46:12.784682 9105 consensus_queue.cc:1035] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:46:12.785342 9297 raft_consensus.cc:2932] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: true } } }
I20240502 20:46:12.785755 6967 raft_consensus.cc:2932] T a9afb75bfc114f59850eaea084a306f0 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: true } } }
I20240502 20:46:12.788148 6831 raft_consensus.cc:2932] T a9afb75bfc114f59850eaea084a306f0 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: true } } }
I20240502 20:46:12.788264 4231 catalog_manager.cc:5484] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c reported cstate change: config changed from index -1 to 2, NON_VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) added. New cstate: current_term: 1 leader_uuid: "7845c397230d49358ba2fad9db53f97c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:12.798134 4374 consensus_queue.cc:237] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } }
I20240502 20:46:12.799321 6703 raft_consensus.cc:1261] T 93b7f7de1f8c49389c3d8c1e03c67f26 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:12.799474 6967 raft_consensus.cc:1261] T 93b7f7de1f8c49389c3d8c1e03c67f26 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 20:46:12.799775 4310 consensus_peers.cc:485] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141): Couldn't send request to peer 923cfab926e7443f8ffd7b21ad0a9d91. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 93b7f7de1f8c49389c3d8c1e03c67f26. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:12.799819 10169 consensus_queue.cc:1035] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:12.799996 10168 consensus_queue.cc:1035] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:12.801892 6138 raft_consensus.cc:2932] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } } }
I20240502 20:46:12.802290 6703 raft_consensus.cc:2932] T 93b7f7de1f8c49389c3d8c1e03c67f26 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } } }
I20240502 20:46:12.803864 6967 raft_consensus.cc:2932] T 93b7f7de1f8c49389c3d8c1e03c67f26 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } } }
I20240502 20:46:12.805464 4227 catalog_manager.cc:5484] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 reported cstate change: config changed from index 4 to 5, NON_VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) added. New cstate: current_term: 1 leader_uuid: "c92e6e46782849479c8a0c6d82276fb2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:12.807264 9908 raft_consensus.cc:979] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd: : Instructing follower 923cfab926e7443f8ffd7b21ad0a9d91 to start an election
I20240502 20:46:12.807353 9908 raft_consensus.cc:1067] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Signalling peer 923cfab926e7443f8ffd7b21ad0a9d91 to start an election
I20240502 20:46:12.807934 4637 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d37fab152a8a44868d3160ec3057bb98"
dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91"
from {username='slave'} at 127.30.32.68:59869
I20240502 20:46:12.808084 4637 raft_consensus.cc:482] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:46:12.808166 4637 raft_consensus.cc:3037] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:12.809055 4637 raft_consensus.cc:504] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } }
I20240502 20:46:12.809517 4637 leader_election.cc:288] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 election: Requested vote from peers ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193), 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
I20240502 20:46:12.809849 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37fab152a8a44868d3160ec3057bb98" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "ed47c023452a497192f35d2f617bf9bd"
I20240502 20:46:12.809965 4768 raft_consensus.cc:3032] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:46:12.810004 4768 raft_consensus.cc:729] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Running, Role: LEADER
I20240502 20:46:12.810184 4768 consensus_queue.cc:260] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } }
I20240502 20:46:12.810452 4768 raft_consensus.cc:3037] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:12.810511 6703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37fab152a8a44868d3160ec3057bb98" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "7845c397230d49358ba2fad9db53f97c"
I20240502 20:46:12.810612 6703 raft_consensus.cc:3037] T d37fab152a8a44868d3160ec3057bb98 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:12.811513 4768 raft_consensus.cc:2454] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 2.
I20240502 20:46:12.811893 6703 raft_consensus.cc:2454] T d37fab152a8a44868d3160ec3057bb98 P 7845c397230d49358ba2fad9db53f97c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 2.
I20240502 20:46:12.812151 4574 leader_election.cc:302] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 7845c397230d49358ba2fad9db53f97c, 923cfab926e7443f8ffd7b21ad0a9d91, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:46:12.812345 7017 raft_consensus.cc:2781] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:46:12.812466 7017 raft_consensus.cc:686] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 2 LEADER]: Becoming Leader. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Running, Role: LEADER
I20240502 20:46:12.812637 7017 consensus_queue.cc:237] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } }
I20240502 20:46:12.814023 6967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37fab152a8a44868d3160ec3057bb98" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "a47e5c56f13843018ac584914a8d4707"
I20240502 20:46:12.814137 6967 raft_consensus.cc:3037] T d37fab152a8a44868d3160ec3057bb98 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:12.814563 4231 catalog_manager.cc:5484] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: term changed from 1 to 2, leader changed from ed47c023452a497192f35d2f617bf9bd (127.30.32.68) to 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67). New cstate: current_term: 2 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:12.815407 6967 raft_consensus.cc:2454] T d37fab152a8a44868d3160ec3057bb98 P a47e5c56f13843018ac584914a8d4707 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 2.
I20240502 20:46:12.816808 6967 consensus_queue.cc:237] T ef4002126e8847809add182f262d62d9 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } }
I20240502 20:46:12.818091 4768 raft_consensus.cc:1261] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer a47e5c56f13843018ac584914a8d4707: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:12.818104 6831 raft_consensus.cc:1261] T ef4002126e8847809add182f262d62d9 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Refusing update from remote peer a47e5c56f13843018ac584914a8d4707: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:12.818486 9156 consensus_queue.cc:1035] T ef4002126e8847809add182f262d62d9 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:12.818704 9775 consensus_queue.cc:1035] T ef4002126e8847809add182f262d62d9 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:12.820358 9156 raft_consensus.cc:2932] T ef4002126e8847809add182f262d62d9 P a47e5c56f13843018ac584914a8d4707 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } } }
I20240502 20:46:12.820534 4507 consensus_queue.cc:237] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } }
I20240502 20:46:12.820747 6831 raft_consensus.cc:2932] T ef4002126e8847809add182f262d62d9 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } } }
I20240502 20:46:12.822077 6835 raft_consensus.cc:1261] T d37730b109d04b6395f75c7552524877 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:12.822803 4222 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ef4002126e8847809add182f262d62d9 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:12.822783 4768 raft_consensus.cc:2932] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } } }
I20240502 20:46:12.823616 8146 consensus_queue.cc:1035] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:12.823719 4231 catalog_manager.cc:5484] T ef4002126e8847809add182f262d62d9 P a47e5c56f13843018ac584914a8d4707 reported cstate change: config changed from index 3 to 4, VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) evicted. New cstate: current_term: 1 leader_uuid: "a47e5c56f13843018ac584914a8d4707" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240502 20:46:12.824098 4441 consensus_peers.cc:485] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d37730b109d04b6395f75c7552524877. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:12.826164 4354 tablet_service.cc:1514] Processing DeleteTablet for tablet ef4002126e8847809add182f262d62d9 with delete_type TABLET_DATA_TOMBSTONED (TS c92e6e46782849479c8a0c6d82276fb2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38306
I20240502 20:46:12.826542 10389 tablet_replica.cc:330] T ef4002126e8847809add182f262d62d9 P c92e6e46782849479c8a0c6d82276fb2: stopping tablet replica
I20240502 20:46:12.826700 10389 raft_consensus.cc:2229] T ef4002126e8847809add182f262d62d9 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:12.826880 10389 raft_consensus.cc:2258] T ef4002126e8847809add182f262d62d9 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:12.827090 8907 raft_consensus.cc:2932] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } } }
I20240502 20:46:12.827514 6835 raft_consensus.cc:2932] T d37730b109d04b6395f75c7552524877 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } } }
I20240502 20:46:12.829939 4227 catalog_manager.cc:5484] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: config changed from index 6 to 7, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:12.830272 10389 ts_tablet_manager.cc:1875] T ef4002126e8847809add182f262d62d9 P c92e6e46782849479c8a0c6d82276fb2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:12.830379 4637 raft_consensus.cc:1261] T d37730b109d04b6395f75c7552524877 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:12.830967 9936 consensus_queue.cc:1035] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:12.834213 10389 ts_tablet_manager.cc:1888] T ef4002126e8847809add182f262d62d9 P c92e6e46782849479c8a0c6d82276fb2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 20:46:12.834301 10389 log.cc:1196] T ef4002126e8847809add182f262d62d9 P c92e6e46782849479c8a0c6d82276fb2: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal/wals/ef4002126e8847809add182f262d62d9
I20240502 20:46:12.834939 4222 catalog_manager.cc:4830] TS c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827): tablet ef4002126e8847809add182f262d62d9 (table rebalancer_extra_table_8 [id=553d578c620843f4b11d547972b8d776]) successfully deleted
I20240502 20:46:12.836395 4637 raft_consensus.cc:2932] T d37730b109d04b6395f75c7552524877 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } } }
I20240502 20:46:12.849958 10217 raft_consensus.cc:482] T 3ba2f5d327f3403ea1604aae9485b960 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:12.850087 10217 raft_consensus.cc:504] T 3ba2f5d327f3403ea1604aae9485b960 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:12.850512 10217 leader_election.cc:288] T 3ba2f5d327f3403ea1604aae9485b960 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:46:12.851106 6967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba2f5d327f3403ea1604aae9485b960" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47e5c56f13843018ac584914a8d4707" is_pre_election: true
I20240502 20:46:12.851168 4374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba2f5d327f3403ea1604aae9485b960" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2" is_pre_election: true
I20240502 20:46:12.851284 6967 raft_consensus.cc:2454] T 3ba2f5d327f3403ea1604aae9485b960 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 0.
I20240502 20:46:12.851306 4374 raft_consensus.cc:2454] T 3ba2f5d327f3403ea1604aae9485b960 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 0.
I20240502 20:46:12.851565 4706 leader_election.cc:302] T 3ba2f5d327f3403ea1604aae9485b960 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:46:12.851931 10217 raft_consensus.cc:2781] T 3ba2f5d327f3403ea1604aae9485b960 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:12.851989 10217 raft_consensus.cc:482] T 3ba2f5d327f3403ea1604aae9485b960 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:12.852031 10217 raft_consensus.cc:3037] T 3ba2f5d327f3403ea1604aae9485b960 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:12.853472 10217 raft_consensus.cc:504] T 3ba2f5d327f3403ea1604aae9485b960 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:12.853881 10217 leader_election.cc:288] T 3ba2f5d327f3403ea1604aae9485b960 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 election: Requested vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:46:12.854538 6967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba2f5d327f3403ea1604aae9485b960" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47e5c56f13843018ac584914a8d4707"
I20240502 20:46:12.854682 6967 raft_consensus.cc:3037] T 3ba2f5d327f3403ea1604aae9485b960 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:12.855561 6967 raft_consensus.cc:2454] T 3ba2f5d327f3403ea1604aae9485b960 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 1.
I20240502 20:46:12.855894 4706 leader_election.cc:302] T 3ba2f5d327f3403ea1604aae9485b960 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a47e5c56f13843018ac584914a8d4707, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:46:12.856068 10217 raft_consensus.cc:2781] T 3ba2f5d327f3403ea1604aae9485b960 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:12.856149 10217 raft_consensus.cc:686] T 3ba2f5d327f3403ea1604aae9485b960 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Becoming Leader. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Running, Role: LEADER
I20240502 20:46:12.856298 10217 consensus_queue.cc:237] T 3ba2f5d327f3403ea1604aae9485b960 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:12.858090 4374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba2f5d327f3403ea1604aae9485b960" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2"
I20240502 20:46:12.858213 4374 raft_consensus.cc:3037] T 3ba2f5d327f3403ea1604aae9485b960 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:12.858520 4235 catalog_manager.cc:5484] T 3ba2f5d327f3403ea1604aae9485b960 P ed47c023452a497192f35d2f617bf9bd reported cstate change: term changed from 0 to 1, leader changed from <none> to ed47c023452a497192f35d2f617bf9bd (127.30.32.68). New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:12.859212 4374 raft_consensus.cc:2454] T 3ba2f5d327f3403ea1604aae9485b960 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 1.
I20240502 20:46:12.862962 7017 raft_consensus.cc:1050] T e7e0539e60064d97a7eb0eacd9df065e P 923cfab926e7443f8ffd7b21ad0a9d91: attempting to promote NON_VOTER a4bf1df7fa6b4948be502c182bfbb833 to VOTER
I20240502 20:46:12.863339 7017 consensus_queue.cc:237] T e7e0539e60064d97a7eb0eacd9df065e P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } }
I20240502 20:46:12.864755 4768 raft_consensus.cc:1261] T e7e0539e60064d97a7eb0eacd9df065e P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:12.864759 6831 raft_consensus.cc:1261] T e7e0539e60064d97a7eb0eacd9df065e P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEARNER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:12.865173 9043 consensus_queue.cc:1035] T e7e0539e60064d97a7eb0eacd9df065e P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:12.865489 7017 consensus_queue.cc:1035] T e7e0539e60064d97a7eb0eacd9df065e P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:12.865760 4507 raft_consensus.cc:1261] T e7e0539e60064d97a7eb0eacd9df065e P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:12.866703 7017 consensus_queue.cc:1035] T e7e0539e60064d97a7eb0eacd9df065e P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:12.873512 9043 raft_consensus.cc:2932] T e7e0539e60064d97a7eb0eacd9df065e P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:12.874115 4507 raft_consensus.cc:2932] T e7e0539e60064d97a7eb0eacd9df065e P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:12.875607 4768 raft_consensus.cc:2932] T e7e0539e60064d97a7eb0eacd9df065e P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:12.876127 6831 raft_consensus.cc:2932] T e7e0539e60064d97a7eb0eacd9df065e P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:12.877240 4235 catalog_manager.cc:5484] T e7e0539e60064d97a7eb0eacd9df065e P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index 4 to 5, a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:12.882133 4637 consensus_queue.cc:237] T e7e0539e60064d97a7eb0eacd9df065e P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } }
I20240502 20:46:12.882196 10415 ts_tablet_manager.cc:902] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91: Initiating tablet copy from peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
I20240502 20:46:12.882751 10415 tablet_copy_client.cc:308] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.69:42429
I20240502 20:46:12.883246 6831 raft_consensus.cc:1261] T e7e0539e60064d97a7eb0eacd9df065e P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:12.883433 4768 raft_consensus.cc:1261] T e7e0539e60064d97a7eb0eacd9df065e P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:12.883406 6723 tablet_copy_service.cc:140] P 7845c397230d49358ba2fad9db53f97c: Received BeginTabletCopySession request for tablet 3d938d01d5154590a9262acd54094ad2 from peer 923cfab926e7443f8ffd7b21ad0a9d91 ({username='slave'} at 127.30.32.67:41973)
I20240502 20:46:12.883535 6723 tablet_copy_service.cc:161] P 7845c397230d49358ba2fad9db53f97c: Beginning new tablet copy session on tablet 3d938d01d5154590a9262acd54094ad2 from peer 923cfab926e7443f8ffd7b21ad0a9d91 at {username='slave'} at 127.30.32.67:41973: session id = 923cfab926e7443f8ffd7b21ad0a9d91-3d938d01d5154590a9262acd54094ad2
I20240502 20:46:12.883596 7017 consensus_queue.cc:1035] T e7e0539e60064d97a7eb0eacd9df065e P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:12.884740 6723 tablet_copy_source_session.cc:205] T 3d938d01d5154590a9262acd54094ad2 P 7845c397230d49358ba2fad9db53f97c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:12.885200 9041 raft_consensus.cc:2932] T e7e0539e60064d97a7eb0eacd9df065e P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:12.885522 6831 raft_consensus.cc:2932] T e7e0539e60064d97a7eb0eacd9df065e P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:12.885596 7017 consensus_queue.cc:1035] T e7e0539e60064d97a7eb0eacd9df065e P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:12.886616 4768 raft_consensus.cc:2932] T e7e0539e60064d97a7eb0eacd9df065e P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } }
I20240502 20:46:12.888958 4219 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e7e0539e60064d97a7eb0eacd9df065e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:12.889369 10415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d938d01d5154590a9262acd54094ad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:12.889469 4231 catalog_manager.cc:5484] T e7e0539e60064d97a7eb0eacd9df065e P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index 5 to 6, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:12.891674 10415 tablet_copy_client.cc:783] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:12.891816 10415 tablet_copy_client.cc:647] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:12.892038 4235 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 20:46:12.893950 4487 tablet_service.cc:1514] Processing DeleteTablet for tablet e7e0539e60064d97a7eb0eacd9df065e with delete_type TABLET_DATA_TOMBSTONED (TS c3d26eafd51547acb780d4aac5ebe7cc not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48890
I20240502 20:46:12.896170 10421 tablet_replica.cc:330] T e7e0539e60064d97a7eb0eacd9df065e P c3d26eafd51547acb780d4aac5ebe7cc: stopping tablet replica
I20240502 20:46:12.896587 10421 raft_consensus.cc:2229] T e7e0539e60064d97a7eb0eacd9df065e P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:12.897135 10421 raft_consensus.cc:2258] T e7e0539e60064d97a7eb0eacd9df065e P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:12.898241 10421 ts_tablet_manager.cc:1875] T e7e0539e60064d97a7eb0eacd9df065e P c3d26eafd51547acb780d4aac5ebe7cc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:12.899252 10415 tablet_copy_client.cc:523] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:12.900642 10415 tablet_bootstrap.cc:492] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:46:12.901031 10421 ts_tablet_manager.cc:1888] T e7e0539e60064d97a7eb0eacd9df065e P c3d26eafd51547acb780d4aac5ebe7cc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:46:12.901099 10421 log.cc:1196] T e7e0539e60064d97a7eb0eacd9df065e P c3d26eafd51547acb780d4aac5ebe7cc: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal/wals/e7e0539e60064d97a7eb0eacd9df065e
I20240502 20:46:12.901712 4221 catalog_manager.cc:4830] TS c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097): tablet e7e0539e60064d97a7eb0eacd9df065e (table rebalance_test_table_15 [id=57f2e4fb3cca4e408bd33dac3f4b9d0c]) successfully deleted
I20240502 20:46:12.904546 10415 tablet_bootstrap.cc:492] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:12.904951 10415 tablet_bootstrap.cc:492] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap complete.
I20240502 20:46:12.905233 10415 ts_tablet_manager.cc:1367] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240502 20:46:12.905735 10415 raft_consensus.cc:348] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } }
I20240502 20:46:12.905965 10415 raft_consensus.cc:729] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: LEARNER
I20240502 20:46:12.906090 10415 consensus_queue.cc:260] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } }
I20240502 20:46:12.906466 10415 ts_tablet_manager.cc:1398] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:12.907433 4637 raft_consensus.cc:1261] T aa5a1d063b7144e1965e4157a7cefca3 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:12.907470 6967 raft_consensus.cc:1261] T aa5a1d063b7144e1965e4157a7cefca3 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:12.907737 6723 tablet_copy_service.cc:342] P 7845c397230d49358ba2fad9db53f97c: Request end of tablet copy session 923cfab926e7443f8ffd7b21ad0a9d91-3d938d01d5154590a9262acd54094ad2 received from {username='slave'} at 127.30.32.67:41973
I20240502 20:46:12.907841 6723 tablet_copy_service.cc:433] P 7845c397230d49358ba2fad9db53f97c: ending tablet copy session 923cfab926e7443f8ffd7b21ad0a9d91-3d938d01d5154590a9262acd54094ad2 on tablet 3d938d01d5154590a9262acd54094ad2 with peer 923cfab926e7443f8ffd7b21ad0a9d91
I20240502 20:46:12.908017 9125 consensus_queue.cc:1035] T aa5a1d063b7144e1965e4157a7cefca3 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:12.908337 10217 consensus_queue.cc:1035] T aa5a1d063b7144e1965e4157a7cefca3 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:12.911120 4227 catalog_manager.cc:5576] TS a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841) does not have the latest schema for tablet aa5a1d063b7144e1965e4157a7cefca3 (table rebalancer_dynamic_table [id=08c4df4d5a16481a9b470989a084c5df]). Expected version 1 got 0
I20240502 20:46:12.912950 10432 tablet.cc:1722] T aa5a1d063b7144e1965e4157a7cefca3 P ed47c023452a497192f35d2f617bf9bd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 20:46:12.913457 10432 tablet.cc:1620] T aa5a1d063b7144e1965e4157a7cefca3 P ed47c023452a497192f35d2f617bf9bd: MemRowSet was empty: no flush needed.
I20240502 20:46:12.914773 9775 raft_consensus.cc:1050] T bcfd626ea0a04fa7b03dd0ede6b35266 P a47e5c56f13843018ac584914a8d4707: attempting to promote NON_VOTER ed47c023452a497192f35d2f617bf9bd to VOTER
I20240502 20:46:12.914857 4228 catalog_manager.cc:5576] TS a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841) does not have the latest schema for tablet aa5a1d063b7144e1965e4157a7cefca3 (table rebalancer_dynamic_table [id=08c4df4d5a16481a9b470989a084c5df]). Expected version 1 got 0
I20240502 20:46:12.915052 10433 tablet.cc:1722] T aa5a1d063b7144e1965e4157a7cefca3 P a47e5c56f13843018ac584914a8d4707: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 20:46:12.915206 9775 consensus_queue.cc:237] T bcfd626ea0a04fa7b03dd0ede6b35266 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } }
I20240502 20:46:12.915342 10433 tablet.cc:1620] T aa5a1d063b7144e1965e4157a7cefca3 P a47e5c56f13843018ac584914a8d4707: MemRowSet was empty: no flush needed.
I20240502 20:46:12.917024 6703 raft_consensus.cc:1261] T bcfd626ea0a04fa7b03dd0ede6b35266 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Refusing update from remote peer a47e5c56f13843018ac584914a8d4707: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:12.917650 9155 consensus_queue.cc:1035] T bcfd626ea0a04fa7b03dd0ede6b35266 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:46:12.918561 7099 raft_consensus.cc:1261] T bcfd626ea0a04fa7b03dd0ede6b35266 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Refusing update from remote peer a47e5c56f13843018ac584914a8d4707: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:12.919322 4768 raft_consensus.cc:1261] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd [term 1 LEARNER]: Refusing update from remote peer a47e5c56f13843018ac584914a8d4707: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:12.921272 9155 consensus_queue.cc:1035] T bcfd626ea0a04fa7b03dd0ede6b35266 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:46:12.921597 4228 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: rebalancer_dynamic_table [id=08c4df4d5a16481a9b470989a084c5df] alter complete (version 1)
I20240502 20:46:12.922190 9156 consensus_queue.cc:1035] T bcfd626ea0a04fa7b03dd0ede6b35266 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:46:12.923295 4235 catalog_manager.cc:5576] TS ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193) does not have the latest schema for tablet aa5a1d063b7144e1965e4157a7cefca3 (table rebalancer_dynamic_table [id=08c4df4d5a16481a9b470989a084c5df]). Expected version 1 got 0
I20240502 20:46:12.924750 10440 tablet.cc:1722] T aa5a1d063b7144e1965e4157a7cefca3 P 923cfab926e7443f8ffd7b21ad0a9d91: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 20:46:12.925161 10440 tablet.cc:1620] T aa5a1d063b7144e1965e4157a7cefca3 P 923cfab926e7443f8ffd7b21ad0a9d91: MemRowSet was empty: no flush needed.
I20240502 20:46:12.929203 4227 catalog_manager.cc:5576] TS 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141) does not have the latest schema for tablet aa5a1d063b7144e1965e4157a7cefca3 (table rebalancer_dynamic_table [id=08c4df4d5a16481a9b470989a084c5df]). Expected version 1 got 0
I20240502 20:46:12.930954 9156 raft_consensus.cc:2932] T bcfd626ea0a04fa7b03dd0ede6b35266 P a47e5c56f13843018ac584914a8d4707 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ed47c023452a497192f35d2f617bf9bd (127.30.32.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } } }
I20240502 20:46:12.932623 4768 raft_consensus.cc:2932] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ed47c023452a497192f35d2f617bf9bd (127.30.32.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } } }
I20240502 20:46:12.933470 6703 raft_consensus.cc:2932] T bcfd626ea0a04fa7b03dd0ede6b35266 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ed47c023452a497192f35d2f617bf9bd (127.30.32.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } } }
I20240502 20:46:12.936292 4227 catalog_manager.cc:5484] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd reported cstate change: config changed from index 2 to 3, ed47c023452a497192f35d2f617bf9bd (127.30.32.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a47e5c56f13843018ac584914a8d4707" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } } }
I20240502 20:46:12.937289 7099 raft_consensus.cc:2932] T bcfd626ea0a04fa7b03dd0ede6b35266 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ed47c023452a497192f35d2f617bf9bd (127.30.32.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } } }
I20240502 20:46:12.939504 6967 consensus_queue.cc:237] T bcfd626ea0a04fa7b03dd0ede6b35266 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } }
I20240502 20:46:12.941046 4769 raft_consensus.cc:1261] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer a47e5c56f13843018ac584914a8d4707: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:12.941107 6703 raft_consensus.cc:1261] T bcfd626ea0a04fa7b03dd0ede6b35266 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Refusing update from remote peer a47e5c56f13843018ac584914a8d4707: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:12.941474 9156 consensus_queue.cc:1035] T bcfd626ea0a04fa7b03dd0ede6b35266 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:12.941715 9775 consensus_queue.cc:1035] T bcfd626ea0a04fa7b03dd0ede6b35266 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:12.943030 9775 raft_consensus.cc:2932] T bcfd626ea0a04fa7b03dd0ede6b35266 P a47e5c56f13843018ac584914a8d4707 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } } }
I20240502 20:46:12.943354 6703 raft_consensus.cc:2932] T bcfd626ea0a04fa7b03dd0ede6b35266 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } } }
I20240502 20:46:12.944190 4769 raft_consensus.cc:2932] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } } }
I20240502 20:46:12.945190 4222 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bcfd626ea0a04fa7b03dd0ede6b35266 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:12.946017 4235 catalog_manager.cc:5484] T bcfd626ea0a04fa7b03dd0ede6b35266 P a47e5c56f13843018ac584914a8d4707 reported cstate change: config changed from index 3 to 4, VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) evicted. New cstate: current_term: 1 leader_uuid: "a47e5c56f13843018ac584914a8d4707" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:12.948709 7079 tablet_service.cc:1514] Processing DeleteTablet for tablet bcfd626ea0a04fa7b03dd0ede6b35266 with delete_type TABLET_DATA_TOMBSTONED (TS 201d2b6485084e7dad2cbb722af94779 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59162
I20240502 20:46:12.949013 10285 tablet_replica.cc:330] T bcfd626ea0a04fa7b03dd0ede6b35266 P 201d2b6485084e7dad2cbb722af94779: stopping tablet replica
I20240502 20:46:12.949178 10285 raft_consensus.cc:2229] T bcfd626ea0a04fa7b03dd0ede6b35266 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:12.949337 10285 raft_consensus.cc:2258] T bcfd626ea0a04fa7b03dd0ede6b35266 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:12.949941 10285 ts_tablet_manager.cc:1875] T bcfd626ea0a04fa7b03dd0ede6b35266 P 201d2b6485084e7dad2cbb722af94779: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:12.951819 10285 ts_tablet_manager.cc:1888] T bcfd626ea0a04fa7b03dd0ede6b35266 P 201d2b6485084e7dad2cbb722af94779: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 20:46:12.951875 10285 log.cc:1196] T bcfd626ea0a04fa7b03dd0ede6b35266 P 201d2b6485084e7dad2cbb722af94779: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/wal/wals/bcfd626ea0a04fa7b03dd0ede6b35266
I20240502 20:46:12.952399 4219 catalog_manager.cc:4830] TS 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): tablet bcfd626ea0a04fa7b03dd0ede6b35266 (table rebalancer_extra_table_0 [id=09a2b90df269425ba3ab97f05a1fc119]) successfully deleted
I20240502 20:46:12.965598 8460 raft_consensus.cc:482] T d78c02773e454ea0b79cf1282eda125b P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:12.965713 8460 raft_consensus.cc:504] T d78c02773e454ea0b79cf1282eda125b P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:12.966133 8460 leader_election.cc:288] T d78c02773e454ea0b79cf1282eda125b P 201d2b6485084e7dad2cbb722af94779 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
I20240502 20:46:12.966491 6967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d78c02773e454ea0b79cf1282eda125b" candidate_uuid: "201d2b6485084e7dad2cbb722af94779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47e5c56f13843018ac584914a8d4707" is_pre_election: true
I20240502 20:46:12.966588 4374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d78c02773e454ea0b79cf1282eda125b" candidate_uuid: "201d2b6485084e7dad2cbb722af94779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2" is_pre_election: true
I20240502 20:46:12.966642 6967 raft_consensus.cc:2454] T d78c02773e454ea0b79cf1282eda125b P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 201d2b6485084e7dad2cbb722af94779 in term 0.
I20240502 20:46:12.966758 4374 raft_consensus.cc:2454] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 201d2b6485084e7dad2cbb722af94779 in term 0.
I20240502 20:46:12.967058 7035 leader_election.cc:302] T d78c02773e454ea0b79cf1282eda125b P 201d2b6485084e7dad2cbb722af94779 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201d2b6485084e7dad2cbb722af94779, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:46:12.967337 8460 raft_consensus.cc:2781] T d78c02773e454ea0b79cf1282eda125b P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:12.967402 8460 raft_consensus.cc:482] T d78c02773e454ea0b79cf1282eda125b P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:12.967459 8460 raft_consensus.cc:3037] T d78c02773e454ea0b79cf1282eda125b P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:12.968328 8460 raft_consensus.cc:504] T d78c02773e454ea0b79cf1282eda125b P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:12.968626 8460 leader_election.cc:288] T d78c02773e454ea0b79cf1282eda125b P 201d2b6485084e7dad2cbb722af94779 [CANDIDATE]: Term 1 election: Requested vote from peers c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
I20240502 20:46:12.968892 4374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d78c02773e454ea0b79cf1282eda125b" candidate_uuid: "201d2b6485084e7dad2cbb722af94779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2"
I20240502 20:46:12.969035 6967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d78c02773e454ea0b79cf1282eda125b" candidate_uuid: "201d2b6485084e7dad2cbb722af94779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47e5c56f13843018ac584914a8d4707"
I20240502 20:46:12.969064 4374 raft_consensus.cc:3037] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:12.969157 6967 raft_consensus.cc:3037] T d78c02773e454ea0b79cf1282eda125b P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:12.969838 4374 raft_consensus.cc:2454] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 201d2b6485084e7dad2cbb722af94779 in term 1.
I20240502 20:46:12.969883 6967 raft_consensus.cc:2454] T d78c02773e454ea0b79cf1282eda125b P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 201d2b6485084e7dad2cbb722af94779 in term 1.
I20240502 20:46:12.970105 7035 leader_election.cc:302] T d78c02773e454ea0b79cf1282eda125b P 201d2b6485084e7dad2cbb722af94779 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201d2b6485084e7dad2cbb722af94779, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:46:12.970820 8460 raft_consensus.cc:2781] T d78c02773e454ea0b79cf1282eda125b P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:12.970918 8460 raft_consensus.cc:686] T d78c02773e454ea0b79cf1282eda125b P 201d2b6485084e7dad2cbb722af94779 [term 1 LEADER]: Becoming Leader. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Running, Role: LEADER
I20240502 20:46:12.971060 8460 consensus_queue.cc:237] T d78c02773e454ea0b79cf1282eda125b P 201d2b6485084e7dad2cbb722af94779 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:12.973640 4227 catalog_manager.cc:5484] T d78c02773e454ea0b79cf1282eda125b P 201d2b6485084e7dad2cbb722af94779 reported cstate change: term changed from 0 to 1, leader changed from <none> to 201d2b6485084e7dad2cbb722af94779 (127.30.32.72). New cstate: current_term: 1 leader_uuid: "201d2b6485084e7dad2cbb722af94779" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:12.996248 4637 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a7ea68e8581a4cd5a6212abc96dfbcde"
dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33732
I20240502 20:46:12.996421 4637 raft_consensus.cc:595] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Received request to transfer leadership
I20240502 20:46:12.998375 6138 raft_consensus.cc:1050] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2: attempting to promote NON_VOTER 7845c397230d49358ba2fad9db53f97c to VOTER
I20240502 20:46:12.998787 6138 consensus_queue.cc:237] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } }
I20240502 20:46:13.000216 4637 raft_consensus.cc:1261] T c9d25657e9a14b35b2f6e8d82f4533c7 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:13.000488 4769 raft_consensus.cc:1261] T c9d25657e9a14b35b2f6e8d82f4533c7 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:13.000792 6703 raft_consensus.cc:1261] T c9d25657e9a14b35b2f6e8d82f4533c7 P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:13.000919 8185 consensus_queue.cc:1035] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:13.001391 10169 consensus_queue.cc:1035] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:13.001618 8185 consensus_queue.cc:1035] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:13.004596 8185 raft_consensus.cc:2932] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:13.005713 6703 raft_consensus.cc:2932] T c9d25657e9a14b35b2f6e8d82f4533c7 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:13.005746 4637 raft_consensus.cc:2932] T c9d25657e9a14b35b2f6e8d82f4533c7 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:13.006783 4769 raft_consensus.cc:2932] T c9d25657e9a14b35b2f6e8d82f4533c7 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:13.009047 4227 catalog_manager.cc:5484] T c9d25657e9a14b35b2f6e8d82f4533c7 P 7845c397230d49358ba2fad9db53f97c reported cstate change: config changed from index 4 to 5, 7845c397230d49358ba2fad9db53f97c (127.30.32.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c92e6e46782849479c8a0c6d82276fb2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:13.012459 4374 consensus_queue.cc:237] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } }
I20240502 20:46:13.013773 6703 raft_consensus.cc:1261] T c9d25657e9a14b35b2f6e8d82f4533c7 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:13.014178 4769 raft_consensus.cc:1261] T c9d25657e9a14b35b2f6e8d82f4533c7 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:13.014174 10169 consensus_queue.cc:1035] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:13.014572 10168 consensus_queue.cc:1035] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:13.016290 10169 raft_consensus.cc:2932] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:13.018915 6703 raft_consensus.cc:2932] T c9d25657e9a14b35b2f6e8d82f4533c7 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:13.020404 4231 catalog_manager.cc:5484] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2 reported cstate change: config changed from index 5 to 6, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New cstate: current_term: 1 leader_uuid: "c92e6e46782849479c8a0c6d82276fb2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:13.021857 4222 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c9d25657e9a14b35b2f6e8d82f4533c7 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:13.022643 4769 raft_consensus.cc:2932] T c9d25657e9a14b35b2f6e8d82f4533c7 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } }
I20240502 20:46:13.023516 4618 tablet_service.cc:1514] Processing DeleteTablet for tablet c9d25657e9a14b35b2f6e8d82f4533c7 with delete_type TABLET_DATA_TOMBSTONED (TS 923cfab926e7443f8ffd7b21ad0a9d91 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57068
I20240502 20:46:13.023808 10225 tablet_replica.cc:330] T c9d25657e9a14b35b2f6e8d82f4533c7 P 923cfab926e7443f8ffd7b21ad0a9d91: stopping tablet replica
I20240502 20:46:13.023989 10225 raft_consensus.cc:2229] T c9d25657e9a14b35b2f6e8d82f4533c7 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:13.024191 10225 raft_consensus.cc:2258] T c9d25657e9a14b35b2f6e8d82f4533c7 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:13.024907 10225 ts_tablet_manager.cc:1875] T c9d25657e9a14b35b2f6e8d82f4533c7 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:13.027261 10225 ts_tablet_manager.cc:1888] T c9d25657e9a14b35b2f6e8d82f4533c7 P 923cfab926e7443f8ffd7b21ad0a9d91: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:46:13.027591 10225 log.cc:1196] T c9d25657e9a14b35b2f6e8d82f4533c7 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal/wals/c9d25657e9a14b35b2f6e8d82f4533c7
I20240502 20:46:13.028148 4219 catalog_manager.cc:4830] TS 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141): tablet c9d25657e9a14b35b2f6e8d82f4533c7 (table rebalance_test_table_13 [id=c22b8adbea7b4dc49a4794a5d8f81f9b]) successfully deleted
I20240502 20:46:13.029168 4228 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 20:46:13.030450 6702 consensus_queue.cc:237] T a135d04d8c3248bc954a07e5b6396fde P 7845c397230d49358ba2fad9db53f97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } }
I20240502 20:46:13.031833 4374 raft_consensus.cc:1261] T a135d04d8c3248bc954a07e5b6396fde P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer 7845c397230d49358ba2fad9db53f97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 20:46:13.032223 6636 consensus_peers.cc:485] T a135d04d8c3248bc954a07e5b6396fde P 7845c397230d49358ba2fad9db53f97c -> Peer 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141): Couldn't send request to peer 923cfab926e7443f8ffd7b21ad0a9d91. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a135d04d8c3248bc954a07e5b6396fde. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:13.032418 7099 raft_consensus.cc:1261] T a135d04d8c3248bc954a07e5b6396fde P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Refusing update from remote peer 7845c397230d49358ba2fad9db53f97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:13.032877 9105 consensus_queue.cc:1035] T a135d04d8c3248bc954a07e5b6396fde P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:46:13.033159 10270 consensus_queue.cc:1035] T a135d04d8c3248bc954a07e5b6396fde P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:46:13.034866 9297 raft_consensus.cc:2932] T a135d04d8c3248bc954a07e5b6396fde P 7845c397230d49358ba2fad9db53f97c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } } }
I20240502 20:46:13.035272 7099 raft_consensus.cc:2932] T a135d04d8c3248bc954a07e5b6396fde P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } } }
I20240502 20:46:13.037068 10432 tablet.cc:1722] T aa5a1d063b7144e1965e4157a7cefca3 P ed47c023452a497192f35d2f617bf9bd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 20:46:13.037425 10432 tablet.cc:1620] T aa5a1d063b7144e1965e4157a7cefca3 P ed47c023452a497192f35d2f617bf9bd: MemRowSet was empty: no flush needed.
I20240502 20:46:13.039196 10433 tablet.cc:1722] T aa5a1d063b7144e1965e4157a7cefca3 P a47e5c56f13843018ac584914a8d4707: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 20:46:13.039530 10433 tablet.cc:1620] T aa5a1d063b7144e1965e4157a7cefca3 P a47e5c56f13843018ac584914a8d4707: MemRowSet was empty: no flush needed.
I20240502 20:46:13.040261 4235 catalog_manager.cc:5484] T a135d04d8c3248bc954a07e5b6396fde P 201d2b6485084e7dad2cbb722af94779 reported cstate change: config changed from index -1 to 2, NON_VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) added. New cstate: current_term: 1 leader_uuid: "7845c397230d49358ba2fad9db53f97c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } } }
I20240502 20:46:13.041934 10440 tablet.cc:1722] T aa5a1d063b7144e1965e4157a7cefca3 P 923cfab926e7443f8ffd7b21ad0a9d91: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 20:46:13.042258 10440 tablet.cc:1620] T aa5a1d063b7144e1965e4157a7cefca3 P 923cfab926e7443f8ffd7b21ad0a9d91: MemRowSet was empty: no flush needed.
I20240502 20:46:13.042800 4374 raft_consensus.cc:2932] T a135d04d8c3248bc954a07e5b6396fde P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } } }
I20240502 20:46:13.046445 4227 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: rebalancer_dynamic_table [id=08c4df4d5a16481a9b470989a084c5df] alter complete (version 2)
I20240502 20:46:13.053932 6702 consensus_queue.cc:237] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: true } }
I20240502 20:46:13.057796 6967 raft_consensus.cc:1261] T 9258667ccc3e4ef8b26fe75531dd4591 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Refusing update from remote peer 7845c397230d49358ba2fad9db53f97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:13.057849 4374 raft_consensus.cc:1261] T 9258667ccc3e4ef8b26fe75531dd4591 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer 7845c397230d49358ba2fad9db53f97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:13.058211 9297 consensus_queue.cc:1035] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:46:13.058380 9105 consensus_queue.cc:1035] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 20:46:13.058775 6639 consensus_peers.cc:485] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c -> Peer ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193): Couldn't send request to peer ed47c023452a497192f35d2f617bf9bd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9258667ccc3e4ef8b26fe75531dd4591. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:13.060247 9297 raft_consensus.cc:2932] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: true } } }
I20240502 20:46:13.060539 4374 raft_consensus.cc:2932] T 9258667ccc3e4ef8b26fe75531dd4591 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: true } } }
I20240502 20:46:13.062769 6957 raft_consensus.cc:2932] T 9258667ccc3e4ef8b26fe75531dd4591 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: true } } }
I20240502 20:46:13.064167 4230 catalog_manager.cc:5484] T 9258667ccc3e4ef8b26fe75531dd4591 P c92e6e46782849479c8a0c6d82276fb2 reported cstate change: config changed from index -1 to 2, NON_VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) added. New cstate: current_term: 1 leader_uuid: "7845c397230d49358ba2fad9db53f97c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: true } } }
I20240502 20:46:13.070596 8236 raft_consensus.cc:979] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91: : Instructing follower c92e6e46782849479c8a0c6d82276fb2 to start an election
I20240502 20:46:13.070698 8236 raft_consensus.cc:1067] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Signalling peer c92e6e46782849479c8a0c6d82276fb2 to start an election
I20240502 20:46:13.071149 4374 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a7ea68e8581a4cd5a6212abc96dfbcde"
dest_uuid: "c92e6e46782849479c8a0c6d82276fb2"
from {username='slave'} at 127.30.32.67:51233
I20240502 20:46:13.071269 4374 raft_consensus.cc:482] T a7ea68e8581a4cd5a6212abc96dfbcde P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:46:13.071321 4374 raft_consensus.cc:3037] T a7ea68e8581a4cd5a6212abc96dfbcde P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:13.072268 4374 raft_consensus.cc:504] T a7ea68e8581a4cd5a6212abc96dfbcde P c92e6e46782849479c8a0c6d82276fb2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } }
I20240502 20:46:13.072743 4374 leader_election.cc:288] T a7ea68e8581a4cd5a6212abc96dfbcde P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 election: Requested vote from peers 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141), ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
I20240502 20:46:13.073212 4769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ea68e8581a4cd5a6212abc96dfbcde" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "ed47c023452a497192f35d2f617bf9bd"
I20240502 20:46:13.073241 4637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ea68e8581a4cd5a6212abc96dfbcde" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91"
I20240502 20:46:13.073354 4769 raft_consensus.cc:3037] T a7ea68e8581a4cd5a6212abc96dfbcde P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:13.073370 4637 raft_consensus.cc:3032] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:46:13.073411 4637 raft_consensus.cc:729] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Running, Role: LEADER
I20240502 20:46:13.073554 4637 consensus_queue.cc:260] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } }
I20240502 20:46:13.073789 4637 raft_consensus.cc:3037] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:13.074034 7098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ea68e8581a4cd5a6212abc96dfbcde" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "201d2b6485084e7dad2cbb722af94779"
I20240502 20:46:13.074157 7098 raft_consensus.cc:3037] T a7ea68e8581a4cd5a6212abc96dfbcde P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:13.074204 4769 raft_consensus.cc:2454] T a7ea68e8581a4cd5a6212abc96dfbcde P ed47c023452a497192f35d2f617bf9bd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 2.
I20240502 20:46:13.074844 4637 raft_consensus.cc:2454] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 2.
I20240502 20:46:13.075155 4310 leader_election.cc:302] T a7ea68e8581a4cd5a6212abc96dfbcde P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, c92e6e46782849479c8a0c6d82276fb2, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:46:13.075229 7098 raft_consensus.cc:2454] T a7ea68e8581a4cd5a6212abc96dfbcde P 201d2b6485084e7dad2cbb722af94779 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 2.
I20240502 20:46:13.075328 10168 raft_consensus.cc:2781] T a7ea68e8581a4cd5a6212abc96dfbcde P c92e6e46782849479c8a0c6d82276fb2 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:46:13.075389 10168 raft_consensus.cc:686] T a7ea68e8581a4cd5a6212abc96dfbcde P c92e6e46782849479c8a0c6d82276fb2 [term 2 LEADER]: Becoming Leader. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Running, Role: LEADER
I20240502 20:46:13.075518 10168 consensus_queue.cc:237] T a7ea68e8581a4cd5a6212abc96dfbcde P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } }
I20240502 20:46:13.077826 4231 catalog_manager.cc:5484] T a7ea68e8581a4cd5a6212abc96dfbcde P c92e6e46782849479c8a0c6d82276fb2 reported cstate change: term changed from 1 to 2, leader changed from 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) to c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65). New cstate: current_term: 2 leader_uuid: "c92e6e46782849479c8a0c6d82276fb2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:13.081662 7098 raft_consensus.cc:1203] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:13.128427 4507 raft_consensus.cc:1203] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 20:46:13.149446 10154 consensus_queue.cc:1035] T 2c67f965350a45f89511679d40f04cbb P 201d2b6485084e7dad2cbb722af94779 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:13.150616 4231 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 20:46:13.153884 10567 consensus_queue.cc:1035] T 2c67f965350a45f89511679d40f04cbb P 201d2b6485084e7dad2cbb722af94779 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:13.160329 10432 tablet.cc:1722] T aa5a1d063b7144e1965e4157a7cefca3 P ed47c023452a497192f35d2f617bf9bd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 20:46:13.160737 10440 tablet.cc:1722] T aa5a1d063b7144e1965e4157a7cefca3 P 923cfab926e7443f8ffd7b21ad0a9d91: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 20:46:13.164758 10433 tablet.cc:1722] T aa5a1d063b7144e1965e4157a7cefca3 P a47e5c56f13843018ac584914a8d4707: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 20:46:13.167071 4230 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: rebalancer_dynamic_table_new_name [id=08c4df4d5a16481a9b470989a084c5df] alter complete (version 3)
I20240502 20:46:13.174083 4637 consensus_queue.cc:237] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } }
W20240502 20:46:13.175573 4575 consensus_peers.cc:485] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 73efeba3d3094ef387e4efb03cff9d4e. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:13.175597 4374 raft_consensus.cc:1261] T 73efeba3d3094ef387e4efb03cff9d4e P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:13.175597 4769 raft_consensus.cc:1261] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:13.176008 7017 consensus_queue.cc:1035] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:13.176194 9043 consensus_queue.cc:1035] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:13.178014 8236 raft_consensus.cc:2932] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } } }
I20240502 20:46:13.178413 4769 raft_consensus.cc:2932] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } } }
I20240502 20:46:13.178596 4374 raft_consensus.cc:2932] T 73efeba3d3094ef387e4efb03cff9d4e P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } } }
I20240502 20:46:13.181444 4230 catalog_manager.cc:5484] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index -1 to 4, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:13.221940 4769 raft_consensus.cc:1261] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd [term 2 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 20:46:13.222677 7017 consensus_queue.cc:1035] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:13.228821 6702 raft_consensus.cc:1261] T d37fab152a8a44868d3160ec3057bb98 P 7845c397230d49358ba2fad9db53f97c [term 2 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 20:46:13.229223 7017 consensus_queue.cc:1035] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:13.237818 6957 raft_consensus.cc:1261] T d37fab152a8a44868d3160ec3057bb98 P a47e5c56f13843018ac584914a8d4707 [term 2 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 20:46:13.238490 8236 consensus_queue.cc:1035] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:13.271855 4231 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 20:46:13.272011 4231 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 20:46:13.274474 4231 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: Sending DeleteTablet for 3 replicas of tablet aa5a1d063b7144e1965e4157a7cefca3
I20240502 20:46:13.275102 4750 tablet_service.cc:1514] Processing DeleteTablet for tablet aa5a1d063b7144e1965e4157a7cefca3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:13 UTC) from {username='slave'} at 127.0.0.1:38526
I20240502 20:46:13.275265 6947 tablet_service.cc:1514] Processing DeleteTablet for tablet aa5a1d063b7144e1965e4157a7cefca3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:13 UTC) from {username='slave'} at 127.0.0.1:45156
I20240502 20:46:13.275434 4618 tablet_service.cc:1514] Processing DeleteTablet for tablet aa5a1d063b7144e1965e4157a7cefca3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:13 UTC) from {username='slave'} at 127.0.0.1:57068
I20240502 20:46:13.275583 10639 tablet_replica.cc:330] T aa5a1d063b7144e1965e4157a7cefca3 P a47e5c56f13843018ac584914a8d4707: stopping tablet replica
I20240502 20:46:13.275606 10225 tablet_replica.cc:330] T aa5a1d063b7144e1965e4157a7cefca3 P 923cfab926e7443f8ffd7b21ad0a9d91: stopping tablet replica
I20240502 20:46:13.275719 10639 raft_consensus.cc:2229] T aa5a1d063b7144e1965e4157a7cefca3 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:13.275749 10225 raft_consensus.cc:2229] T aa5a1d063b7144e1965e4157a7cefca3 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:13.275826 10639 raft_consensus.cc:2258] T aa5a1d063b7144e1965e4157a7cefca3 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:13.275879 10225 raft_consensus.cc:2258] T aa5a1d063b7144e1965e4157a7cefca3 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:13.276737 10225 ts_tablet_manager.cc:1875] T aa5a1d063b7144e1965e4157a7cefca3 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:13.277501 10639 ts_tablet_manager.cc:1875] T aa5a1d063b7144e1965e4157a7cefca3 P a47e5c56f13843018ac584914a8d4707: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:13.278052 10638 tablet_replica.cc:330] T aa5a1d063b7144e1965e4157a7cefca3 P ed47c023452a497192f35d2f617bf9bd: stopping tablet replica
I20240502 20:46:13.278216 10638 raft_consensus.cc:2229] T aa5a1d063b7144e1965e4157a7cefca3 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:13.278443 10638 raft_consensus.cc:2258] T aa5a1d063b7144e1965e4157a7cefca3 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:13.278841 10225 ts_tablet_manager.cc:1888] T aa5a1d063b7144e1965e4157a7cefca3 P 923cfab926e7443f8ffd7b21ad0a9d91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:13.278913 10225 log.cc:1196] T aa5a1d063b7144e1965e4157a7cefca3 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal/wals/aa5a1d063b7144e1965e4157a7cefca3
I20240502 20:46:13.279139 10638 ts_tablet_manager.cc:1875] T aa5a1d063b7144e1965e4157a7cefca3 P ed47c023452a497192f35d2f617bf9bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:13.279181 10225 ts_tablet_manager.cc:1909] T aa5a1d063b7144e1965e4157a7cefca3 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting consensus metadata
I20240502 20:46:13.280051 4219 catalog_manager.cc:4830] TS 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141): tablet aa5a1d063b7144e1965e4157a7cefca3 (table rebalancer_dynamic_table_new_name [id=08c4df4d5a16481a9b470989a084c5df]) successfully deleted
I20240502 20:46:13.280562 10639 ts_tablet_manager.cc:1888] T aa5a1d063b7144e1965e4157a7cefca3 P a47e5c56f13843018ac584914a8d4707: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:13.280648 10639 log.cc:1196] T aa5a1d063b7144e1965e4157a7cefca3 P a47e5c56f13843018ac584914a8d4707: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/wal/wals/aa5a1d063b7144e1965e4157a7cefca3
I20240502 20:46:13.280929 10639 ts_tablet_manager.cc:1909] T aa5a1d063b7144e1965e4157a7cefca3 P a47e5c56f13843018ac584914a8d4707: Deleting consensus metadata
I20240502 20:46:13.281666 10638 ts_tablet_manager.cc:1888] T aa5a1d063b7144e1965e4157a7cefca3 P ed47c023452a497192f35d2f617bf9bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:13.281700 4222 catalog_manager.cc:4830] TS a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): tablet aa5a1d063b7144e1965e4157a7cefca3 (table rebalancer_dynamic_table_new_name [id=08c4df4d5a16481a9b470989a084c5df]) successfully deleted
I20240502 20:46:13.281749 10638 log.cc:1196] T aa5a1d063b7144e1965e4157a7cefca3 P ed47c023452a497192f35d2f617bf9bd: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal/wals/aa5a1d063b7144e1965e4157a7cefca3
I20240502 20:46:13.282043 10638 ts_tablet_manager.cc:1909] T aa5a1d063b7144e1965e4157a7cefca3 P ed47c023452a497192f35d2f617bf9bd: Deleting consensus metadata
I20240502 20:46:13.282652 4222 catalog_manager.cc:4830] TS ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193): tablet aa5a1d063b7144e1965e4157a7cefca3 (table rebalancer_dynamic_table_new_name [id=08c4df4d5a16481a9b470989a084c5df]) successfully deleted
I20240502 20:46:13.312034 10640 ts_tablet_manager.cc:902] T a9afb75bfc114f59850eaea084a306f0 P ed47c023452a497192f35d2f617bf9bd: Initiating tablet copy from peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
I20240502 20:46:13.312578 10640 tablet_copy_client.cc:308] T a9afb75bfc114f59850eaea084a306f0 P ed47c023452a497192f35d2f617bf9bd: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.69:42429
I20240502 20:46:13.313136 6723 tablet_copy_service.cc:140] P 7845c397230d49358ba2fad9db53f97c: Received BeginTabletCopySession request for tablet a9afb75bfc114f59850eaea084a306f0 from peer ed47c023452a497192f35d2f617bf9bd ({username='slave'} at 127.30.32.68:47929)
I20240502 20:46:13.313256 6723 tablet_copy_service.cc:161] P 7845c397230d49358ba2fad9db53f97c: Beginning new tablet copy session on tablet a9afb75bfc114f59850eaea084a306f0 from peer ed47c023452a497192f35d2f617bf9bd at {username='slave'} at 127.30.32.68:47929: session id = ed47c023452a497192f35d2f617bf9bd-a9afb75bfc114f59850eaea084a306f0
I20240502 20:46:13.314018 6723 tablet_copy_source_session.cc:205] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:13.314945 10640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9afb75bfc114f59850eaea084a306f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:13.316923 10640 tablet_copy_client.cc:783] T a9afb75bfc114f59850eaea084a306f0 P ed47c023452a497192f35d2f617bf9bd: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:13.317188 10640 tablet_copy_client.cc:647] T a9afb75bfc114f59850eaea084a306f0 P ed47c023452a497192f35d2f617bf9bd: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:13.318235 10640 tablet_copy_client.cc:523] T a9afb75bfc114f59850eaea084a306f0 P ed47c023452a497192f35d2f617bf9bd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:13.319228 10640 tablet_bootstrap.cc:492] T a9afb75bfc114f59850eaea084a306f0 P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:46:13.321926 10640 tablet_bootstrap.cc:492] T a9afb75bfc114f59850eaea084a306f0 P ed47c023452a497192f35d2f617bf9bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:13.322147 10640 tablet_bootstrap.cc:492] T a9afb75bfc114f59850eaea084a306f0 P ed47c023452a497192f35d2f617bf9bd: Bootstrap complete.
I20240502 20:46:13.322301 10640 ts_tablet_manager.cc:1367] T a9afb75bfc114f59850eaea084a306f0 P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240502 20:46:13.322675 10640 raft_consensus.cc:348] T a9afb75bfc114f59850eaea084a306f0 P ed47c023452a497192f35d2f617bf9bd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: true } }
I20240502 20:46:13.322777 10640 raft_consensus.cc:729] T a9afb75bfc114f59850eaea084a306f0 P ed47c023452a497192f35d2f617bf9bd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: LEARNER
I20240502 20:46:13.322877 10640 consensus_queue.cc:260] T a9afb75bfc114f59850eaea084a306f0 P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: true } }
I20240502 20:46:13.323143 10640 ts_tablet_manager.cc:1398] T a9afb75bfc114f59850eaea084a306f0 P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:13.323596 6723 tablet_copy_service.cc:342] P 7845c397230d49358ba2fad9db53f97c: Request end of tablet copy session ed47c023452a497192f35d2f617bf9bd-a9afb75bfc114f59850eaea084a306f0 received from {username='slave'} at 127.30.32.68:47929
I20240502 20:46:13.323668 6723 tablet_copy_service.cc:433] P 7845c397230d49358ba2fad9db53f97c: ending tablet copy session ed47c023452a497192f35d2f617bf9bd-a9afb75bfc114f59850eaea084a306f0 on tablet a9afb75bfc114f59850eaea084a306f0 with peer ed47c023452a497192f35d2f617bf9bd
I20240502 20:46:13.353440 10646 ts_tablet_manager.cc:902] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2: Initiating tablet copy from peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
I20240502 20:46:13.354097 10646 tablet_copy_client.cc:308] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.69:42429
I20240502 20:46:13.354646 6723 tablet_copy_service.cc:140] P 7845c397230d49358ba2fad9db53f97c: Received BeginTabletCopySession request for tablet ac57ad308b174217ac74c4cb5e1c7285 from peer c92e6e46782849479c8a0c6d82276fb2 ({username='slave'} at 127.30.32.65:34313)
I20240502 20:46:13.354753 6723 tablet_copy_service.cc:161] P 7845c397230d49358ba2fad9db53f97c: Beginning new tablet copy session on tablet ac57ad308b174217ac74c4cb5e1c7285 from peer c92e6e46782849479c8a0c6d82276fb2 at {username='slave'} at 127.30.32.65:34313: session id = c92e6e46782849479c8a0c6d82276fb2-ac57ad308b174217ac74c4cb5e1c7285
I20240502 20:46:13.355496 6723 tablet_copy_source_session.cc:205] T ac57ad308b174217ac74c4cb5e1c7285 P 7845c397230d49358ba2fad9db53f97c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:13.356642 10646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac57ad308b174217ac74c4cb5e1c7285. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:13.358872 10646 tablet_copy_client.cc:783] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:13.359052 10646 tablet_copy_client.cc:647] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:13.360452 10646 tablet_copy_client.cc:523] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:13.362110 10646 tablet_bootstrap.cc:492] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:46:13.365944 10646 tablet_bootstrap.cc:492] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:13.366163 10646 tablet_bootstrap.cc:492] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2: Bootstrap complete.
I20240502 20:46:13.366314 10646 ts_tablet_manager.cc:1367] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.002s
I20240502 20:46:13.366665 10646 raft_consensus.cc:348] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true } }
I20240502 20:46:13.366765 10646 raft_consensus.cc:729] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: LEARNER
I20240502 20:46:13.366844 10646 consensus_queue.cc:260] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true } }
I20240502 20:46:13.367086 10646 ts_tablet_manager.cc:1398] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:13.367600 6723 tablet_copy_service.cc:342] P 7845c397230d49358ba2fad9db53f97c: Request end of tablet copy session c92e6e46782849479c8a0c6d82276fb2-ac57ad308b174217ac74c4cb5e1c7285 received from {username='slave'} at 127.30.32.65:34313
I20240502 20:46:13.367688 6723 tablet_copy_service.cc:433] P 7845c397230d49358ba2fad9db53f97c: ending tablet copy session c92e6e46782849479c8a0c6d82276fb2-ac57ad308b174217ac74c4cb5e1c7285 on tablet ac57ad308b174217ac74c4cb5e1c7285 with peer c92e6e46782849479c8a0c6d82276fb2
I20240502 20:46:13.377194 4231 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41812:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 20:46:13.381879 7079 tablet_service.cc:1467] Processing CreateTablet for tablet fd0c4a2a86604d39973dc7278c3a45a8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=28389bfda85a41a6b72a550afc0ef67d]), partition=
I20240502 20:46:13.381984 4750 tablet_service.cc:1467] Processing CreateTablet for tablet fd0c4a2a86604d39973dc7278c3a45a8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=28389bfda85a41a6b72a550afc0ef67d]), partition=
I20240502 20:46:13.382164 7079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd0c4a2a86604d39973dc7278c3a45a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:13.382216 4750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd0c4a2a86604d39973dc7278c3a45a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:13.384161 10652 tablet_bootstrap.cc:492] T fd0c4a2a86604d39973dc7278c3a45a8 P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:46:13.384549 6683 tablet_service.cc:1467] Processing CreateTablet for tablet fd0c4a2a86604d39973dc7278c3a45a8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=28389bfda85a41a6b72a550afc0ef67d]), partition=
I20240502 20:46:13.384799 6683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd0c4a2a86604d39973dc7278c3a45a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:13.385151 10652 tablet_bootstrap.cc:654] T fd0c4a2a86604d39973dc7278c3a45a8 P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:13.386278 10652 tablet_bootstrap.cc:492] T fd0c4a2a86604d39973dc7278c3a45a8 P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:46:13.386348 10652 ts_tablet_manager.cc:1367] T fd0c4a2a86604d39973dc7278c3a45a8 P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:13.386736 10652 raft_consensus.cc:348] T fd0c4a2a86604d39973dc7278c3a45a8 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:13.386902 10652 raft_consensus.cc:374] T fd0c4a2a86604d39973dc7278c3a45a8 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:13.386960 10652 raft_consensus.cc:729] T fd0c4a2a86604d39973dc7278c3a45a8 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:46:13.387084 10652 consensus_queue.cc:260] T fd0c4a2a86604d39973dc7278c3a45a8 P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:13.387135 10653 tablet_bootstrap.cc:492] T fd0c4a2a86604d39973dc7278c3a45a8 P 201d2b6485084e7dad2cbb722af94779: Bootstrap starting.
I20240502 20:46:13.387920 10652 ts_tablet_manager.cc:1398] T fd0c4a2a86604d39973dc7278c3a45a8 P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:13.388012 10653 tablet_bootstrap.cc:654] T fd0c4a2a86604d39973dc7278c3a45a8 P 201d2b6485084e7dad2cbb722af94779: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:13.389174 10653 tablet_bootstrap.cc:492] T fd0c4a2a86604d39973dc7278c3a45a8 P 201d2b6485084e7dad2cbb722af94779: No bootstrap required, opened a new log
I20240502 20:46:13.389276 10653 ts_tablet_manager.cc:1367] T fd0c4a2a86604d39973dc7278c3a45a8 P 201d2b6485084e7dad2cbb722af94779: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:13.390177 10653 raft_consensus.cc:348] T fd0c4a2a86604d39973dc7278c3a45a8 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:13.390293 10653 raft_consensus.cc:374] T fd0c4a2a86604d39973dc7278c3a45a8 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:13.390369 10653 raft_consensus.cc:729] T fd0c4a2a86604d39973dc7278c3a45a8 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Initialized, Role: FOLLOWER
I20240502 20:46:13.390522 10653 consensus_queue.cc:260] T fd0c4a2a86604d39973dc7278c3a45a8 P 201d2b6485084e7dad2cbb722af94779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:13.390918 10653 ts_tablet_manager.cc:1398] T fd0c4a2a86604d39973dc7278c3a45a8 P 201d2b6485084e7dad2cbb722af94779: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:13.394780 10657 ts_tablet_manager.cc:902] T d37730b109d04b6395f75c7552524877 P 201d2b6485084e7dad2cbb722af94779: Initiating tablet copy from peer c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097)
I20240502 20:46:13.395324 10657 tablet_copy_client.cc:308] T d37730b109d04b6395f75c7552524877 P 201d2b6485084e7dad2cbb722af94779: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.66:42097
I20240502 20:46:13.396009 4528 tablet_copy_service.cc:140] P c3d26eafd51547acb780d4aac5ebe7cc: Received BeginTabletCopySession request for tablet d37730b109d04b6395f75c7552524877 from peer 201d2b6485084e7dad2cbb722af94779 ({username='slave'} at 127.30.32.72:54159)
I20240502 20:46:13.396136 4528 tablet_copy_service.cc:161] P c3d26eafd51547acb780d4aac5ebe7cc: Beginning new tablet copy session on tablet d37730b109d04b6395f75c7552524877 from peer 201d2b6485084e7dad2cbb722af94779 at {username='slave'} at 127.30.32.72:54159: session id = 201d2b6485084e7dad2cbb722af94779-d37730b109d04b6395f75c7552524877
I20240502 20:46:13.396857 4528 tablet_copy_source_session.cc:205] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:13.397187 10655 tablet_bootstrap.cc:492] T fd0c4a2a86604d39973dc7278c3a45a8 P 7845c397230d49358ba2fad9db53f97c: Bootstrap starting.
I20240502 20:46:13.397851 10657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d37730b109d04b6395f75c7552524877. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:13.398170 10655 tablet_bootstrap.cc:654] T fd0c4a2a86604d39973dc7278c3a45a8 P 7845c397230d49358ba2fad9db53f97c: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:13.399403 10655 tablet_bootstrap.cc:492] T fd0c4a2a86604d39973dc7278c3a45a8 P 7845c397230d49358ba2fad9db53f97c: No bootstrap required, opened a new log
I20240502 20:46:13.399492 10655 ts_tablet_manager.cc:1367] T fd0c4a2a86604d39973dc7278c3a45a8 P 7845c397230d49358ba2fad9db53f97c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:13.399894 10657 tablet_copy_client.cc:783] T d37730b109d04b6395f75c7552524877 P 201d2b6485084e7dad2cbb722af94779: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:13.399878 10655 raft_consensus.cc:348] T fd0c4a2a86604d39973dc7278c3a45a8 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:13.400025 10655 raft_consensus.cc:374] T fd0c4a2a86604d39973dc7278c3a45a8 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:13.400070 10655 raft_consensus.cc:729] T fd0c4a2a86604d39973dc7278c3a45a8 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Initialized, Role: FOLLOWER
I20240502 20:46:13.400193 10657 tablet_copy_client.cc:647] T d37730b109d04b6395f75c7552524877 P 201d2b6485084e7dad2cbb722af94779: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:13.400229 10655 consensus_queue.cc:260] T fd0c4a2a86604d39973dc7278c3a45a8 P 7845c397230d49358ba2fad9db53f97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:13.400547 10655 ts_tablet_manager.cc:1398] T fd0c4a2a86604d39973dc7278c3a45a8 P 7845c397230d49358ba2fad9db53f97c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:13.402423 10657 tablet_copy_client.cc:523] T d37730b109d04b6395f75c7552524877 P 201d2b6485084e7dad2cbb722af94779: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:13.403609 10657 tablet_bootstrap.cc:492] T d37730b109d04b6395f75c7552524877 P 201d2b6485084e7dad2cbb722af94779: Bootstrap starting.
I20240502 20:46:13.408417 10415 ts_tablet_manager.cc:902] T 93b7f7de1f8c49389c3d8c1e03c67f26 P 923cfab926e7443f8ffd7b21ad0a9d91: Initiating tablet copy from peer c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:46:13.408862 10415 tablet_copy_client.cc:308] T 93b7f7de1f8c49389c3d8c1e03c67f26 P 923cfab926e7443f8ffd7b21ad0a9d91: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.65:33827
I20240502 20:46:13.414448 10668 ts_tablet_manager.cc:902] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91: Initiating tablet copy from peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
I20240502 20:46:13.414623 4397 tablet_copy_service.cc:140] P c92e6e46782849479c8a0c6d82276fb2: Received BeginTabletCopySession request for tablet 93b7f7de1f8c49389c3d8c1e03c67f26 from peer 923cfab926e7443f8ffd7b21ad0a9d91 ({username='slave'} at 127.30.32.67:51233)
I20240502 20:46:13.414733 4397 tablet_copy_service.cc:161] P c92e6e46782849479c8a0c6d82276fb2: Beginning new tablet copy session on tablet 93b7f7de1f8c49389c3d8c1e03c67f26 from peer 923cfab926e7443f8ffd7b21ad0a9d91 at {username='slave'} at 127.30.32.67:51233: session id = 923cfab926e7443f8ffd7b21ad0a9d91-93b7f7de1f8c49389c3d8c1e03c67f26
I20240502 20:46:13.415051 10668 tablet_copy_client.cc:308] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.69:42429
I20240502 20:46:13.415467 6723 tablet_copy_service.cc:140] P 7845c397230d49358ba2fad9db53f97c: Received BeginTabletCopySession request for tablet a135d04d8c3248bc954a07e5b6396fde from peer 923cfab926e7443f8ffd7b21ad0a9d91 ({username='slave'} at 127.30.32.67:41973)
I20240502 20:46:13.415542 6723 tablet_copy_service.cc:161] P 7845c397230d49358ba2fad9db53f97c: Beginning new tablet copy session on tablet a135d04d8c3248bc954a07e5b6396fde from peer 923cfab926e7443f8ffd7b21ad0a9d91 at {username='slave'} at 127.30.32.67:41973: session id = 923cfab926e7443f8ffd7b21ad0a9d91-a135d04d8c3248bc954a07e5b6396fde
I20240502 20:46:13.415597 4397 tablet_copy_source_session.cc:205] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:13.416379 6723 tablet_copy_source_session.cc:205] T a135d04d8c3248bc954a07e5b6396fde P 7845c397230d49358ba2fad9db53f97c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:13.416523 10415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93b7f7de1f8c49389c3d8c1e03c67f26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:13.418514 10415 tablet_copy_client.cc:783] T 93b7f7de1f8c49389c3d8c1e03c67f26 P 923cfab926e7443f8ffd7b21ad0a9d91: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:13.418696 10415 tablet_copy_client.cc:647] T 93b7f7de1f8c49389c3d8c1e03c67f26 P 923cfab926e7443f8ffd7b21ad0a9d91: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:13.420826 10668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a135d04d8c3248bc954a07e5b6396fde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:13.421708 10415 tablet_copy_client.cc:523] T 93b7f7de1f8c49389c3d8c1e03c67f26 P 923cfab926e7443f8ffd7b21ad0a9d91: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:13.423014 10415 tablet_bootstrap.cc:492] T 93b7f7de1f8c49389c3d8c1e03c67f26 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:46:13.423229 10668 tablet_copy_client.cc:783] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:13.423414 10668 tablet_copy_client.cc:647] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:13.425969 10668 tablet_copy_client.cc:523] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:13.426944 10668 tablet_bootstrap.cc:492] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:46:13.430905 10415 tablet_bootstrap.cc:492] T 93b7f7de1f8c49389c3d8c1e03c67f26 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:13.431155 10415 tablet_bootstrap.cc:492] T 93b7f7de1f8c49389c3d8c1e03c67f26 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap complete.
I20240502 20:46:13.431309 10415 ts_tablet_manager.cc:1367] T 93b7f7de1f8c49389c3d8c1e03c67f26 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240502 20:46:13.431706 10415 raft_consensus.cc:348] T 93b7f7de1f8c49389c3d8c1e03c67f26 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } }
I20240502 20:46:13.431808 10415 raft_consensus.cc:729] T 93b7f7de1f8c49389c3d8c1e03c67f26 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: LEARNER
I20240502 20:46:13.431932 10415 consensus_queue.cc:260] T 93b7f7de1f8c49389c3d8c1e03c67f26 P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } }
I20240502 20:46:13.432178 10415 ts_tablet_manager.cc:1398] T 93b7f7de1f8c49389c3d8c1e03c67f26 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 20:46:13.432811 4397 tablet_copy_service.cc:342] P c92e6e46782849479c8a0c6d82276fb2: Request end of tablet copy session 923cfab926e7443f8ffd7b21ad0a9d91-93b7f7de1f8c49389c3d8c1e03c67f26 received from {username='slave'} at 127.30.32.67:51233
I20240502 20:46:13.432888 4397 tablet_copy_service.cc:433] P c92e6e46782849479c8a0c6d82276fb2: ending tablet copy session 923cfab926e7443f8ffd7b21ad0a9d91-93b7f7de1f8c49389c3d8c1e03c67f26 on tablet 93b7f7de1f8c49389c3d8c1e03c67f26 with peer 923cfab926e7443f8ffd7b21ad0a9d91
I20240502 20:46:13.434473 10154 raft_consensus.cc:482] T fd0c4a2a86604d39973dc7278c3a45a8 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:13.434567 10154 raft_consensus.cc:504] T fd0c4a2a86604d39973dc7278c3a45a8 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:13.434746 10657 tablet_bootstrap.cc:492] T d37730b109d04b6395f75c7552524877 P 201d2b6485084e7dad2cbb722af94779: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:13.435065 10657 tablet_bootstrap.cc:492] T d37730b109d04b6395f75c7552524877 P 201d2b6485084e7dad2cbb722af94779: Bootstrap complete.
I20240502 20:46:13.435045 10154 leader_election.cc:288] T fd0c4a2a86604d39973dc7278c3a45a8 P 201d2b6485084e7dad2cbb722af94779 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193), 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
I20240502 20:46:13.435312 10657 ts_tablet_manager.cc:1367] T d37730b109d04b6395f75c7552524877 P 201d2b6485084e7dad2cbb722af94779: Time spent bootstrapping tablet: real 0.032s user 0.027s sys 0.000s
I20240502 20:46:13.435595 4769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0c4a2a86604d39973dc7278c3a45a8" candidate_uuid: "201d2b6485084e7dad2cbb722af94779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd" is_pre_election: true
I20240502 20:46:13.435685 6702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0c4a2a86604d39973dc7278c3a45a8" candidate_uuid: "201d2b6485084e7dad2cbb722af94779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7845c397230d49358ba2fad9db53f97c" is_pre_election: true
I20240502 20:46:13.435724 4769 raft_consensus.cc:2454] T fd0c4a2a86604d39973dc7278c3a45a8 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 201d2b6485084e7dad2cbb722af94779 in term 0.
I20240502 20:46:13.435748 10657 raft_consensus.cc:348] T d37730b109d04b6395f75c7552524877 P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } }
I20240502 20:46:13.435815 6702 raft_consensus.cc:2454] T fd0c4a2a86604d39973dc7278c3a45a8 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 201d2b6485084e7dad2cbb722af94779 in term 0.
I20240502 20:46:13.435853 10657 raft_consensus.cc:729] T d37730b109d04b6395f75c7552524877 P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Initialized, Role: LEARNER
I20240502 20:46:13.435945 10657 consensus_queue.cc:260] T d37730b109d04b6395f75c7552524877 P 201d2b6485084e7dad2cbb722af94779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } }
I20240502 20:46:13.436151 7035 leader_election.cc:302] T fd0c4a2a86604d39973dc7278c3a45a8 P 201d2b6485084e7dad2cbb722af94779 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201d2b6485084e7dad2cbb722af94779, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:46:13.436305 10668 tablet_bootstrap.cc:492] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:13.436301 10154 raft_consensus.cc:2781] T fd0c4a2a86604d39973dc7278c3a45a8 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:13.436344 10154 raft_consensus.cc:482] T fd0c4a2a86604d39973dc7278c3a45a8 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:13.436380 10154 raft_consensus.cc:3037] T fd0c4a2a86604d39973dc7278c3a45a8 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:13.436511 10668 tablet_bootstrap.cc:492] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap complete.
I20240502 20:46:13.436627 10668 ts_tablet_manager.cc:1367] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240502 20:46:13.436906 10657 ts_tablet_manager.cc:1398] T d37730b109d04b6395f75c7552524877 P 201d2b6485084e7dad2cbb722af94779: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:13.437356 10154 raft_consensus.cc:504] T fd0c4a2a86604d39973dc7278c3a45a8 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:13.437657 10154 leader_election.cc:288] T fd0c4a2a86604d39973dc7278c3a45a8 P 201d2b6485084e7dad2cbb722af94779 [CANDIDATE]: Term 1 election: Requested vote from peers ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193), 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
I20240502 20:46:13.438022 4528 tablet_copy_service.cc:342] P c3d26eafd51547acb780d4aac5ebe7cc: Request end of tablet copy session 201d2b6485084e7dad2cbb722af94779-d37730b109d04b6395f75c7552524877 received from {username='slave'} at 127.30.32.72:54159
I20240502 20:46:13.438129 4528 tablet_copy_service.cc:433] P c3d26eafd51547acb780d4aac5ebe7cc: ending tablet copy session 201d2b6485084e7dad2cbb722af94779-d37730b109d04b6395f75c7552524877 on tablet d37730b109d04b6395f75c7552524877 with peer 201d2b6485084e7dad2cbb722af94779
I20240502 20:46:13.438127 4769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0c4a2a86604d39973dc7278c3a45a8" candidate_uuid: "201d2b6485084e7dad2cbb722af94779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd"
I20240502 20:46:13.438235 4769 raft_consensus.cc:3037] T fd0c4a2a86604d39973dc7278c3a45a8 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:13.438357 6702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0c4a2a86604d39973dc7278c3a45a8" candidate_uuid: "201d2b6485084e7dad2cbb722af94779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7845c397230d49358ba2fad9db53f97c"
I20240502 20:46:13.438469 6702 raft_consensus.cc:3037] T fd0c4a2a86604d39973dc7278c3a45a8 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:13.438558 10668 raft_consensus.cc:348] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } }
I20240502 20:46:13.438688 10668 raft_consensus.cc:729] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: LEARNER
I20240502 20:46:13.438794 10668 consensus_queue.cc:260] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } }
I20240502 20:46:13.439231 4769 raft_consensus.cc:2454] T fd0c4a2a86604d39973dc7278c3a45a8 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 201d2b6485084e7dad2cbb722af94779 in term 1.
I20240502 20:46:13.439306 10668 ts_tablet_manager.cc:1398] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:46:13.439466 6702 raft_consensus.cc:2454] T fd0c4a2a86604d39973dc7278c3a45a8 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 201d2b6485084e7dad2cbb722af94779 in term 1.
I20240502 20:46:13.439795 6723 tablet_copy_service.cc:342] P 7845c397230d49358ba2fad9db53f97c: Request end of tablet copy session 923cfab926e7443f8ffd7b21ad0a9d91-a135d04d8c3248bc954a07e5b6396fde received from {username='slave'} at 127.30.32.67:41973
I20240502 20:46:13.439855 6723 tablet_copy_service.cc:433] P 7845c397230d49358ba2fad9db53f97c: ending tablet copy session 923cfab926e7443f8ffd7b21ad0a9d91-a135d04d8c3248bc954a07e5b6396fde on tablet a135d04d8c3248bc954a07e5b6396fde with peer 923cfab926e7443f8ffd7b21ad0a9d91
I20240502 20:46:13.440305 7034 leader_election.cc:302] T fd0c4a2a86604d39973dc7278c3a45a8 P 201d2b6485084e7dad2cbb722af94779 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201d2b6485084e7dad2cbb722af94779, 7845c397230d49358ba2fad9db53f97c; no voters:
I20240502 20:46:13.440860 10154 raft_consensus.cc:2781] T fd0c4a2a86604d39973dc7278c3a45a8 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:13.440960 10154 raft_consensus.cc:686] T fd0c4a2a86604d39973dc7278c3a45a8 P 201d2b6485084e7dad2cbb722af94779 [term 1 LEADER]: Becoming Leader. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Running, Role: LEADER
I20240502 20:46:13.441115 10154 consensus_queue.cc:237] T fd0c4a2a86604d39973dc7278c3a45a8 P 201d2b6485084e7dad2cbb722af94779 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:13.443498 4231 catalog_manager.cc:5484] T fd0c4a2a86604d39973dc7278c3a45a8 P 201d2b6485084e7dad2cbb722af94779 reported cstate change: term changed from 0 to 1, leader changed from <none> to 201d2b6485084e7dad2cbb722af94779 (127.30.32.72). New cstate: current_term: 1 leader_uuid: "201d2b6485084e7dad2cbb722af94779" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:13.454879 9125 consensus_queue.cc:1035] T 3ba2f5d327f3403ea1604aae9485b960 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:13.456791 10217 consensus_queue.cc:1035] T 3ba2f5d327f3403ea1604aae9485b960 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:13.481002 4769 raft_consensus.cc:1261] T a7ea68e8581a4cd5a6212abc96dfbcde P ed47c023452a497192f35d2f617bf9bd [term 2 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 20:46:13.481246 8460 consensus_queue.cc:1035] T d78c02773e454ea0b79cf1282eda125b P 201d2b6485084e7dad2cbb722af94779 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:13.481515 8185 consensus_queue.cc:1035] T a7ea68e8581a4cd5a6212abc96dfbcde P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:13.486398 10154 consensus_queue.cc:1035] T d78c02773e454ea0b79cf1282eda125b P 201d2b6485084e7dad2cbb722af94779 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:13.490953 4637 raft_consensus.cc:1203] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 20:46:13.542912 4235 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41812:
name: "rebalancer_extra_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:46:13.549486 6815 tablet_service.cc:1467] Processing CreateTablet for tablet b2508a5aa1c14b63971895953e149e17 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=55b8fc4db6a644ebbe0e3156af5bd7e2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:13.549822 6815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2508a5aa1c14b63971895953e149e17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:13.550181 6947 tablet_service.cc:1467] Processing CreateTablet for tablet b2508a5aa1c14b63971895953e149e17 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=55b8fc4db6a644ebbe0e3156af5bd7e2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:13.550431 6947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2508a5aa1c14b63971895953e149e17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:13.551986 10742 tablet_bootstrap.cc:492] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap starting.
I20240502 20:46:13.552482 6945 tablet_service.cc:1467] Processing CreateTablet for tablet b69500b0144a42f59383b8c5d38324b3 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=55b8fc4db6a644ebbe0e3156af5bd7e2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:13.552489 4230 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 20:46:13.552752 6945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b69500b0144a42f59383b8c5d38324b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:13.552744 6815 tablet_service.cc:1467] Processing CreateTablet for tablet 71537846373844cc9ef396b79af60db4 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=55b8fc4db6a644ebbe0e3156af5bd7e2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:13.553020 6815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71537846373844cc9ef396b79af60db4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:13.553095 10743 tablet_bootstrap.cc:492] T b2508a5aa1c14b63971895953e149e17 P a47e5c56f13843018ac584914a8d4707: Bootstrap starting.
I20240502 20:46:13.554333 10743 tablet_bootstrap.cc:654] T b2508a5aa1c14b63971895953e149e17 P a47e5c56f13843018ac584914a8d4707: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:13.555130 7079 tablet_service.cc:1467] Processing CreateTablet for tablet b2508a5aa1c14b63971895953e149e17 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=55b8fc4db6a644ebbe0e3156af5bd7e2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:13.555411 7079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2508a5aa1c14b63971895953e149e17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:13.555527 6683 tablet_service.cc:1467] Processing CreateTablet for tablet b69500b0144a42f59383b8c5d38324b3 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=55b8fc4db6a644ebbe0e3156af5bd7e2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:13.555788 6683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b69500b0144a42f59383b8c5d38324b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:13.556349 6681 tablet_service.cc:1467] Processing CreateTablet for tablet 71537846373844cc9ef396b79af60db4 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=55b8fc4db6a644ebbe0e3156af5bd7e2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:13.556572 6681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71537846373844cc9ef396b79af60db4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:13.556738 10742 tablet_bootstrap.cc:654] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:13.557801 10743 tablet_bootstrap.cc:492] T b2508a5aa1c14b63971895953e149e17 P a47e5c56f13843018ac584914a8d4707: No bootstrap required, opened a new log
I20240502 20:46:13.557801 7078 tablet_service.cc:1467] Processing CreateTablet for tablet b69500b0144a42f59383b8c5d38324b3 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=55b8fc4db6a644ebbe0e3156af5bd7e2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:13.557886 10743 ts_tablet_manager.cc:1367] T b2508a5aa1c14b63971895953e149e17 P a47e5c56f13843018ac584914a8d4707: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 20:46:13.558005 10655 tablet_bootstrap.cc:492] T b69500b0144a42f59383b8c5d38324b3 P 7845c397230d49358ba2fad9db53f97c: Bootstrap starting.
I20240502 20:46:13.558037 7078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b69500b0144a42f59383b8c5d38324b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:13.558326 10743 raft_consensus.cc:348] T b2508a5aa1c14b63971895953e149e17 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:13.558426 10743 raft_consensus.cc:374] T b2508a5aa1c14b63971895953e149e17 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:13.558458 10743 raft_consensus.cc:729] T b2508a5aa1c14b63971895953e149e17 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Initialized, Role: FOLLOWER
I20240502 20:46:13.558599 10743 consensus_queue.cc:260] T b2508a5aa1c14b63971895953e149e17 P a47e5c56f13843018ac584914a8d4707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:13.558912 10655 tablet_bootstrap.cc:654] T b69500b0144a42f59383b8c5d38324b3 P 7845c397230d49358ba2fad9db53f97c: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:13.558933 10743 ts_tablet_manager.cc:1398] T b2508a5aa1c14b63971895953e149e17 P a47e5c56f13843018ac584914a8d4707: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:13.559101 10743 tablet_bootstrap.cc:492] T b69500b0144a42f59383b8c5d38324b3 P a47e5c56f13843018ac584914a8d4707: Bootstrap starting.
I20240502 20:46:13.559736 7077 tablet_service.cc:1467] Processing CreateTablet for tablet 71537846373844cc9ef396b79af60db4 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=55b8fc4db6a644ebbe0e3156af5bd7e2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:13.559973 10743 tablet_bootstrap.cc:654] T b69500b0144a42f59383b8c5d38324b3 P a47e5c56f13843018ac584914a8d4707: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:13.559984 7077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71537846373844cc9ef396b79af60db4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:13.560051 10655 tablet_bootstrap.cc:492] T b69500b0144a42f59383b8c5d38324b3 P 7845c397230d49358ba2fad9db53f97c: No bootstrap required, opened a new log
I20240502 20:46:13.560093 10653 tablet_bootstrap.cc:492] T b2508a5aa1c14b63971895953e149e17 P 201d2b6485084e7dad2cbb722af94779: Bootstrap starting.
I20240502 20:46:13.560140 10655 ts_tablet_manager.cc:1367] T b69500b0144a42f59383b8c5d38324b3 P 7845c397230d49358ba2fad9db53f97c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:13.560570 10655 raft_consensus.cc:348] T b69500b0144a42f59383b8c5d38324b3 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:13.560663 10655 raft_consensus.cc:374] T b69500b0144a42f59383b8c5d38324b3 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:13.560694 10655 raft_consensus.cc:729] T b69500b0144a42f59383b8c5d38324b3 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Initialized, Role: FOLLOWER
I20240502 20:46:13.560829 10655 consensus_queue.cc:260] T b69500b0144a42f59383b8c5d38324b3 P 7845c397230d49358ba2fad9db53f97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:13.560953 10653 tablet_bootstrap.cc:654] T b2508a5aa1c14b63971895953e149e17 P 201d2b6485084e7dad2cbb722af94779: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:13.561193 10655 ts_tablet_manager.cc:1398] T b69500b0144a42f59383b8c5d38324b3 P 7845c397230d49358ba2fad9db53f97c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:13.561371 10655 tablet_bootstrap.cc:492] T 71537846373844cc9ef396b79af60db4 P 7845c397230d49358ba2fad9db53f97c: Bootstrap starting.
I20240502 20:46:13.562014 10653 tablet_bootstrap.cc:492] T b2508a5aa1c14b63971895953e149e17 P 201d2b6485084e7dad2cbb722af94779: No bootstrap required, opened a new log
I20240502 20:46:13.562093 10653 ts_tablet_manager.cc:1367] T b2508a5aa1c14b63971895953e149e17 P 201d2b6485084e7dad2cbb722af94779: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 20:46:13.562196 10655 tablet_bootstrap.cc:654] T 71537846373844cc9ef396b79af60db4 P 7845c397230d49358ba2fad9db53f97c: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:13.562500 10653 raft_consensus.cc:348] T b2508a5aa1c14b63971895953e149e17 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:13.562608 10653 raft_consensus.cc:374] T b2508a5aa1c14b63971895953e149e17 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:13.562651 10653 raft_consensus.cc:729] T b2508a5aa1c14b63971895953e149e17 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Initialized, Role: FOLLOWER
I20240502 20:46:13.562788 10653 consensus_queue.cc:260] T b2508a5aa1c14b63971895953e149e17 P 201d2b6485084e7dad2cbb722af94779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:13.563081 10653 ts_tablet_manager.cc:1398] T b2508a5aa1c14b63971895953e149e17 P 201d2b6485084e7dad2cbb722af94779: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:13.563239 10653 tablet_bootstrap.cc:492] T b69500b0144a42f59383b8c5d38324b3 P 201d2b6485084e7dad2cbb722af94779: Bootstrap starting.
I20240502 20:46:13.564544 4769 raft_consensus.cc:1261] T fd0c4a2a86604d39973dc7278c3a45a8 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer 201d2b6485084e7dad2cbb722af94779: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:13.566000 10653 tablet_bootstrap.cc:654] T b69500b0144a42f59383b8c5d38324b3 P 201d2b6485084e7dad2cbb722af94779: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:13.566021 6702 raft_consensus.cc:1261] T fd0c4a2a86604d39973dc7278c3a45a8 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Refusing update from remote peer 201d2b6485084e7dad2cbb722af94779: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:13.566565 8460 consensus_queue.cc:1035] T fd0c4a2a86604d39973dc7278c3a45a8 P 201d2b6485084e7dad2cbb722af94779 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:13.567267 10743 tablet_bootstrap.cc:492] T b69500b0144a42f59383b8c5d38324b3 P a47e5c56f13843018ac584914a8d4707: No bootstrap required, opened a new log
I20240502 20:46:13.567355 10743 ts_tablet_manager.cc:1367] T b69500b0144a42f59383b8c5d38324b3 P a47e5c56f13843018ac584914a8d4707: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 20:46:13.567497 10567 consensus_queue.cc:1035] T fd0c4a2a86604d39973dc7278c3a45a8 P 201d2b6485084e7dad2cbb722af94779 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:13.567798 10743 raft_consensus.cc:348] T b69500b0144a42f59383b8c5d38324b3 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:13.567899 10743 raft_consensus.cc:374] T b69500b0144a42f59383b8c5d38324b3 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:13.567934 10743 raft_consensus.cc:729] T b69500b0144a42f59383b8c5d38324b3 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Initialized, Role: FOLLOWER
I20240502 20:46:13.568044 10743 consensus_queue.cc:260] T b69500b0144a42f59383b8c5d38324b3 P a47e5c56f13843018ac584914a8d4707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:13.568604 9043 raft_consensus.cc:1050] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91: attempting to promote NON_VOTER 201d2b6485084e7dad2cbb722af94779 to VOTER
I20240502 20:46:13.569023 10743 ts_tablet_manager.cc:1398] T b69500b0144a42f59383b8c5d38324b3 P a47e5c56f13843018ac584914a8d4707: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:13.569049 9043 consensus_queue.cc:237] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } }
I20240502 20:46:13.570501 4507 raft_consensus.cc:1261] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:13.570736 4374 raft_consensus.cc:1261] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:13.571090 9041 consensus_queue.cc:1035] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:13.571177 4235 catalog_manager.cc:5576] TS ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193) does not have the latest schema for tablet fd0c4a2a86604d39973dc7278c3a45a8 (table rebalancer_dynamic_table [id=28389bfda85a41a6b72a550afc0ef67d]). Expected version 1 got 0
I20240502 20:46:13.571414 9041 consensus_queue.cc:1035] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:13.572456 10655 tablet_bootstrap.cc:492] T 71537846373844cc9ef396b79af60db4 P 7845c397230d49358ba2fad9db53f97c: No bootstrap required, opened a new log
I20240502 20:46:13.572542 10655 ts_tablet_manager.cc:1367] T 71537846373844cc9ef396b79af60db4 P 7845c397230d49358ba2fad9db53f97c: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240502 20:46:13.573743 10655 raft_consensus.cc:348] T 71537846373844cc9ef396b79af60db4 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:13.573911 10655 raft_consensus.cc:374] T 71537846373844cc9ef396b79af60db4 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:13.573987 10655 raft_consensus.cc:729] T 71537846373844cc9ef396b79af60db4 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Initialized, Role: FOLLOWER
I20240502 20:46:13.574275 4230 catalog_manager.cc:5576] TS 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429) does not have the latest schema for tablet fd0c4a2a86604d39973dc7278c3a45a8 (table rebalancer_dynamic_table [id=28389bfda85a41a6b72a550afc0ef67d]). Expected version 1 got 0
I20240502 20:46:13.574352 7098 raft_consensus.cc:1261] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:13.574761 9042 consensus_queue.cc:1035] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:13.574795 10655 consensus_queue.cc:260] T 71537846373844cc9ef396b79af60db4 P 7845c397230d49358ba2fad9db53f97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:13.575228 10655 ts_tablet_manager.cc:1398] T 71537846373844cc9ef396b79af60db4 P 7845c397230d49358ba2fad9db53f97c: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 20:46:13.579028 4230 catalog_manager.cc:5576] TS 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643) does not have the latest schema for tablet fd0c4a2a86604d39973dc7278c3a45a8 (table rebalancer_dynamic_table [id=28389bfda85a41a6b72a550afc0ef67d]). Expected version 1 got 0
I20240502 20:46:13.580893 10432 tablet.cc:1722] T fd0c4a2a86604d39973dc7278c3a45a8 P ed47c023452a497192f35d2f617bf9bd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 20:46:13.580829 4230 catalog_manager.cc:5576] TS 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429) does not have the latest schema for tablet fd0c4a2a86604d39973dc7278c3a45a8 (table rebalancer_dynamic_table [id=28389bfda85a41a6b72a550afc0ef67d]). Expected version 1 got 0
I20240502 20:46:13.581252 10432 tablet.cc:1620] T fd0c4a2a86604d39973dc7278c3a45a8 P ed47c023452a497192f35d2f617bf9bd: MemRowSet was empty: no flush needed.
I20240502 20:46:13.581401 4227 catalog_manager.cc:5576] TS 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643) does not have the latest schema for tablet fd0c4a2a86604d39973dc7278c3a45a8 (table rebalancer_dynamic_table [id=28389bfda85a41a6b72a550afc0ef67d]). Expected version 1 got 0
I20240502 20:46:13.583736 9042 raft_consensus.cc:2932] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:13.584142 4227 catalog_manager.cc:5576] TS ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193) does not have the latest schema for tablet fd0c4a2a86604d39973dc7278c3a45a8 (table rebalancer_dynamic_table [id=28389bfda85a41a6b72a550afc0ef67d]). Expected version 1 got 0
I20240502 20:46:13.585492 10757 tablet.cc:1722] T fd0c4a2a86604d39973dc7278c3a45a8 P 201d2b6485084e7dad2cbb722af94779: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 20:46:13.585867 10757 tablet.cc:1620] T fd0c4a2a86604d39973dc7278c3a45a8 P 201d2b6485084e7dad2cbb722af94779: MemRowSet was empty: no flush needed.
I20240502 20:46:13.585983 4374 raft_consensus.cc:2932] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:13.586431 10758 tablet.cc:1722] T fd0c4a2a86604d39973dc7278c3a45a8 P 7845c397230d49358ba2fad9db53f97c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 20:46:13.586756 10758 tablet.cc:1620] T fd0c4a2a86604d39973dc7278c3a45a8 P 7845c397230d49358ba2fad9db53f97c: MemRowSet was empty: no flush needed.
I20240502 20:46:13.589285 4231 catalog_manager.cc:5484] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P c92e6e46782849479c8a0c6d82276fb2 reported cstate change: config changed from index 4 to 5, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:13.589511 4235 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: rebalancer_dynamic_table [id=28389bfda85a41a6b72a550afc0ef67d] alter complete (version 1)
I20240502 20:46:13.592267 7098 raft_consensus.cc:2932] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:13.597705 10653 tablet_bootstrap.cc:492] T b69500b0144a42f59383b8c5d38324b3 P 201d2b6485084e7dad2cbb722af94779: No bootstrap required, opened a new log
I20240502 20:46:13.597818 10653 ts_tablet_manager.cc:1367] T b69500b0144a42f59383b8c5d38324b3 P 201d2b6485084e7dad2cbb722af94779: Time spent bootstrapping tablet: real 0.035s user 0.000s sys 0.002s
I20240502 20:46:13.598243 10653 raft_consensus.cc:348] T b69500b0144a42f59383b8c5d38324b3 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:13.598341 10653 raft_consensus.cc:374] T b69500b0144a42f59383b8c5d38324b3 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:13.598364 10653 raft_consensus.cc:729] T b69500b0144a42f59383b8c5d38324b3 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Initialized, Role: FOLLOWER
I20240502 20:46:13.598474 10653 consensus_queue.cc:260] T b69500b0144a42f59383b8c5d38324b3 P 201d2b6485084e7dad2cbb722af94779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:13.598726 10653 ts_tablet_manager.cc:1398] T b69500b0144a42f59383b8c5d38324b3 P 201d2b6485084e7dad2cbb722af94779: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:13.598858 10653 tablet_bootstrap.cc:492] T 71537846373844cc9ef396b79af60db4 P 201d2b6485084e7dad2cbb722af94779: Bootstrap starting.
I20240502 20:46:13.600085 4507 raft_consensus.cc:2932] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:13.600314 4637 consensus_queue.cc:237] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } }
I20240502 20:46:13.603878 4374 raft_consensus.cc:1261] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:13.604663 7098 raft_consensus.cc:1261] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:13.605062 8236 consensus_queue.cc:1035] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:13.606587 8236 consensus_queue.cc:1035] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:13.608330 7017 raft_consensus.cc:2932] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:13.608767 7098 raft_consensus.cc:2932] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:13.610266 4219 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 76b6dee18d224e879bf0d3b6bbb2b9b5 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:13.611946 4227 catalog_manager.cc:5484] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 201d2b6485084e7dad2cbb722af94779 reported cstate change: config changed from index 5 to 6, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:13.612413 10653 tablet_bootstrap.cc:654] T 71537846373844cc9ef396b79af60db4 P 201d2b6485084e7dad2cbb722af94779: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:13.614851 4374 raft_consensus.cc:2932] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:13.616794 10653 tablet_bootstrap.cc:492] T 71537846373844cc9ef396b79af60db4 P 201d2b6485084e7dad2cbb722af94779: No bootstrap required, opened a new log
I20240502 20:46:13.616927 10653 ts_tablet_manager.cc:1367] T 71537846373844cc9ef396b79af60db4 P 201d2b6485084e7dad2cbb722af94779: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.001s
I20240502 20:46:13.617672 10653 raft_consensus.cc:348] T 71537846373844cc9ef396b79af60db4 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:13.618054 10653 raft_consensus.cc:374] T 71537846373844cc9ef396b79af60db4 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:13.618139 10653 raft_consensus.cc:729] T 71537846373844cc9ef396b79af60db4 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Initialized, Role: FOLLOWER
I20240502 20:46:13.618315 10653 consensus_queue.cc:260] T 71537846373844cc9ef396b79af60db4 P 201d2b6485084e7dad2cbb722af94779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:13.618693 10653 ts_tablet_manager.cc:1398] T 71537846373844cc9ef396b79af60db4 P 201d2b6485084e7dad2cbb722af94779: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 20:46:13.619947 10742 tablet_bootstrap.cc:492] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833: No bootstrap required, opened a new log
I20240502 20:46:13.620056 10742 ts_tablet_manager.cc:1367] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833: Time spent bootstrapping tablet: real 0.068s user 0.002s sys 0.000s
I20240502 20:46:13.620461 10742 raft_consensus.cc:348] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:13.620565 10742 raft_consensus.cc:374] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:13.620600 10742 raft_consensus.cc:729] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Initialized, Role: FOLLOWER
I20240502 20:46:13.620719 10742 consensus_queue.cc:260] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:13.621049 10742 ts_tablet_manager.cc:1398] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:13.621244 10742 tablet_bootstrap.cc:492] T 71537846373844cc9ef396b79af60db4 P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap starting.
I20240502 20:46:13.622138 10742 tablet_bootstrap.cc:654] T 71537846373844cc9ef396b79af60db4 P a4bf1df7fa6b4948be502c182bfbb833: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:13.623195 10742 tablet_bootstrap.cc:492] T 71537846373844cc9ef396b79af60db4 P a4bf1df7fa6b4948be502c182bfbb833: No bootstrap required, opened a new log
I20240502 20:46:13.623282 10742 ts_tablet_manager.cc:1367] T 71537846373844cc9ef396b79af60db4 P a4bf1df7fa6b4948be502c182bfbb833: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:13.623634 10742 raft_consensus.cc:348] T 71537846373844cc9ef396b79af60db4 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:13.623723 10742 raft_consensus.cc:374] T 71537846373844cc9ef396b79af60db4 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:13.623759 10742 raft_consensus.cc:729] T 71537846373844cc9ef396b79af60db4 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Initialized, Role: FOLLOWER
I20240502 20:46:13.623888 10742 consensus_queue.cc:260] T 71537846373844cc9ef396b79af60db4 P a4bf1df7fa6b4948be502c182bfbb833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:13.624140 10742 ts_tablet_manager.cc:1398] T 71537846373844cc9ef396b79af60db4 P a4bf1df7fa6b4948be502c182bfbb833: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:13.626991 4487 tablet_service.cc:1514] Processing DeleteTablet for tablet 76b6dee18d224e879bf0d3b6bbb2b9b5 with delete_type TABLET_DATA_TOMBSTONED (TS c3d26eafd51547acb780d4aac5ebe7cc not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48890
I20240502 20:46:13.627419 10761 tablet_replica.cc:330] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P c3d26eafd51547acb780d4aac5ebe7cc: stopping tablet replica
I20240502 20:46:13.627591 10761 raft_consensus.cc:2229] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:13.627875 10761 raft_consensus.cc:2258] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:13.628523 10761 ts_tablet_manager.cc:1875] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:13.631072 10761 ts_tablet_manager.cc:1888] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P c3d26eafd51547acb780d4aac5ebe7cc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:46:13.631134 10761 log.cc:1196] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal/wals/76b6dee18d224e879bf0d3b6bbb2b9b5
I20240502 20:46:13.631601 4221 catalog_manager.cc:4830] TS c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097): tablet 76b6dee18d224e879bf0d3b6bbb2b9b5 (table rebalance_test_table_1 [id=3a26d60c6de746e891d970c2ce9c55cc]) successfully deleted
I20240502 20:46:13.637787 10640 ts_tablet_manager.cc:902] T 9258667ccc3e4ef8b26fe75531dd4591 P ed47c023452a497192f35d2f617bf9bd: Initiating tablet copy from peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
I20240502 20:46:13.638278 10640 tablet_copy_client.cc:308] T 9258667ccc3e4ef8b26fe75531dd4591 P ed47c023452a497192f35d2f617bf9bd: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.69:42429
I20240502 20:46:13.638805 6723 tablet_copy_service.cc:140] P 7845c397230d49358ba2fad9db53f97c: Received BeginTabletCopySession request for tablet 9258667ccc3e4ef8b26fe75531dd4591 from peer ed47c023452a497192f35d2f617bf9bd ({username='slave'} at 127.30.32.68:47929)
I20240502 20:46:13.638901 6723 tablet_copy_service.cc:161] P 7845c397230d49358ba2fad9db53f97c: Beginning new tablet copy session on tablet 9258667ccc3e4ef8b26fe75531dd4591 from peer ed47c023452a497192f35d2f617bf9bd at {username='slave'} at 127.30.32.68:47929: session id = ed47c023452a497192f35d2f617bf9bd-9258667ccc3e4ef8b26fe75531dd4591
I20240502 20:46:13.639639 6723 tablet_copy_source_session.cc:205] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:13.640450 10640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9258667ccc3e4ef8b26fe75531dd4591. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:13.642257 10640 tablet_copy_client.cc:783] T 9258667ccc3e4ef8b26fe75531dd4591 P ed47c023452a497192f35d2f617bf9bd: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:13.642406 10640 tablet_copy_client.cc:647] T 9258667ccc3e4ef8b26fe75531dd4591 P ed47c023452a497192f35d2f617bf9bd: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:13.643497 10640 tablet_copy_client.cc:523] T 9258667ccc3e4ef8b26fe75531dd4591 P ed47c023452a497192f35d2f617bf9bd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:13.644526 10640 tablet_bootstrap.cc:492] T 9258667ccc3e4ef8b26fe75531dd4591 P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:46:13.644647 7098 raft_consensus.cc:1261] T a7ea68e8581a4cd5a6212abc96dfbcde P 201d2b6485084e7dad2cbb722af94779 [term 2 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 20:46:13.645149 8185 consensus_queue.cc:1035] T a7ea68e8581a4cd5a6212abc96dfbcde P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:13.648730 4637 raft_consensus.cc:1261] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91 [term 2 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 20:46:13.649457 8185 consensus_queue.cc:1035] T a7ea68e8581a4cd5a6212abc96dfbcde P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:13.650025 10640 tablet_bootstrap.cc:492] T 9258667ccc3e4ef8b26fe75531dd4591 P ed47c023452a497192f35d2f617bf9bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:13.650283 10640 tablet_bootstrap.cc:492] T 9258667ccc3e4ef8b26fe75531dd4591 P ed47c023452a497192f35d2f617bf9bd: Bootstrap complete.
I20240502 20:46:13.650446 10640 ts_tablet_manager.cc:1367] T 9258667ccc3e4ef8b26fe75531dd4591 P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240502 20:46:13.650907 10640 raft_consensus.cc:348] T 9258667ccc3e4ef8b26fe75531dd4591 P ed47c023452a497192f35d2f617bf9bd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: true } }
I20240502 20:46:13.651018 10640 raft_consensus.cc:729] T 9258667ccc3e4ef8b26fe75531dd4591 P ed47c023452a497192f35d2f617bf9bd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: LEARNER
I20240502 20:46:13.651090 10640 consensus_queue.cc:260] T 9258667ccc3e4ef8b26fe75531dd4591 P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: true } }
I20240502 20:46:13.651329 10640 ts_tablet_manager.cc:1398] T 9258667ccc3e4ef8b26fe75531dd4591 P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:13.651397 4374 consensus_queue.cc:237] T a7ea68e8581a4cd5a6212abc96dfbcde P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } }
I20240502 20:46:13.652428 6723 tablet_copy_service.cc:342] P 7845c397230d49358ba2fad9db53f97c: Request end of tablet copy session ed47c023452a497192f35d2f617bf9bd-9258667ccc3e4ef8b26fe75531dd4591 received from {username='slave'} at 127.30.32.68:47929
I20240502 20:46:13.652493 6723 tablet_copy_service.cc:433] P 7845c397230d49358ba2fad9db53f97c: ending tablet copy session ed47c023452a497192f35d2f617bf9bd-9258667ccc3e4ef8b26fe75531dd4591 on tablet 9258667ccc3e4ef8b26fe75531dd4591 with peer ed47c023452a497192f35d2f617bf9bd
I20240502 20:46:13.653234 7098 raft_consensus.cc:1261] T a7ea68e8581a4cd5a6212abc96dfbcde P 201d2b6485084e7dad2cbb722af94779 [term 2 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 20:46:13.654559 10769 consensus_queue.cc:1035] T a7ea68e8581a4cd5a6212abc96dfbcde P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:13.654863 4769 raft_consensus.cc:1261] T a7ea68e8581a4cd5a6212abc96dfbcde P ed47c023452a497192f35d2f617bf9bd [term 2 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 20:46:13.655263 10769 consensus_queue.cc:1035] T a7ea68e8581a4cd5a6212abc96dfbcde P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:13.656168 8185 raft_consensus.cc:2932] T a7ea68e8581a4cd5a6212abc96dfbcde P c92e6e46782849479c8a0c6d82276fb2 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:13.656592 7099 raft_consensus.cc:2932] T a7ea68e8581a4cd5a6212abc96dfbcde P 201d2b6485084e7dad2cbb722af94779 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:13.657378 4769 raft_consensus.cc:2932] T a7ea68e8581a4cd5a6212abc96dfbcde P ed47c023452a497192f35d2f617bf9bd [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:13.657972 4222 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a7ea68e8581a4cd5a6212abc96dfbcde with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:13.660490 4231 catalog_manager.cc:5484] T a7ea68e8581a4cd5a6212abc96dfbcde P ed47c023452a497192f35d2f617bf9bd reported cstate change: config changed from index 5 to 7, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New cstate: current_term: 2 leader_uuid: "c92e6e46782849479c8a0c6d82276fb2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:13.661779 4637 consensus_queue.cc:237] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } }
I20240502 20:46:13.663158 6967 raft_consensus.cc:1261] T d37fab152a8a44868d3160ec3057bb98 P a47e5c56f13843018ac584914a8d4707 [term 2 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 20:46:13.663476 6702 raft_consensus.cc:1261] T d37fab152a8a44868d3160ec3057bb98 P 7845c397230d49358ba2fad9db53f97c [term 2 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 20:46:13.663687 4618 tablet_service.cc:1514] Processing DeleteTablet for tablet a7ea68e8581a4cd5a6212abc96dfbcde with delete_type TABLET_DATA_TOMBSTONED (TS 923cfab926e7443f8ffd7b21ad0a9d91 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57068
I20240502 20:46:13.663910 10225 tablet_replica.cc:330] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91: stopping tablet replica
I20240502 20:46:13.663919 9041 consensus_queue.cc:1035] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:46:13.664079 10225 raft_consensus.cc:2229] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:13.664162 10225 raft_consensus.cc:2258] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:13.664093 8236 consensus_queue.cc:1035] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:46:13.664798 10225 ts_tablet_manager.cc:1875] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:13.666010 9041 raft_consensus.cc:2932] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:13.666379 6967 raft_consensus.cc:2932] T d37fab152a8a44868d3160ec3057bb98 P a47e5c56f13843018ac584914a8d4707 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:13.667089 10225 ts_tablet_manager.cc:1888] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 20:46:13.667155 10225 log.cc:1196] T a7ea68e8581a4cd5a6212abc96dfbcde P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal/wals/a7ea68e8581a4cd5a6212abc96dfbcde
I20240502 20:46:13.668807 4230 catalog_manager.cc:5484] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index 8 to 10, VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) evicted. New cstate: current_term: 2 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:13.671249 4750 tablet_service.cc:1514] Processing DeleteTablet for tablet d37fab152a8a44868d3160ec3057bb98 with delete_type TABLET_DATA_TOMBSTONED (TS ed47c023452a497192f35d2f617bf9bd not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:38526
I20240502 20:46:13.671490 10638 tablet_replica.cc:330] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd: stopping tablet replica
I20240502 20:46:13.671638 10638 raft_consensus.cc:2229] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:13.671738 10638 raft_consensus.cc:2258] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:13.672353 10638 ts_tablet_manager.cc:1875] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:13.673194 4219 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d37fab152a8a44868d3160ec3057bb98 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:13.673372 4219 catalog_manager.cc:4830] TS 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141): tablet a7ea68e8581a4cd5a6212abc96dfbcde (table rebalance_test_table_17 [id=75dfa602f9634de99d6f55e4fdeea21c]) successfully deleted
I20240502 20:46:13.673378 6702 raft_consensus.cc:2932] T d37fab152a8a44868d3160ec3057bb98 P 7845c397230d49358ba2fad9db53f97c [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:13.674685 10638 ts_tablet_manager.cc:1888] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 20:46:13.674765 10638 log.cc:1196] T d37fab152a8a44868d3160ec3057bb98 P ed47c023452a497192f35d2f617bf9bd: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal/wals/d37fab152a8a44868d3160ec3057bb98
I20240502 20:46:13.675350 4222 catalog_manager.cc:4830] TS ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193): tablet d37fab152a8a44868d3160ec3057bb98 (table rebalance_test_table_5 [id=5c60c96389ff4cf3bbd5edb6a694c70d]) successfully deleted
I20240502 20:46:13.690866 4230 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 20:46:13.696959 10757 tablet.cc:1722] T fd0c4a2a86604d39973dc7278c3a45a8 P 201d2b6485084e7dad2cbb722af94779: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 20:46:13.697307 10757 tablet.cc:1620] T fd0c4a2a86604d39973dc7278c3a45a8 P 201d2b6485084e7dad2cbb722af94779: MemRowSet was empty: no flush needed.
I20240502 20:46:13.697391 10758 tablet.cc:1722] T fd0c4a2a86604d39973dc7278c3a45a8 P 7845c397230d49358ba2fad9db53f97c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 20:46:13.697650 10758 tablet.cc:1620] T fd0c4a2a86604d39973dc7278c3a45a8 P 7845c397230d49358ba2fad9db53f97c: MemRowSet was empty: no flush needed.
I20240502 20:46:13.700392 9908 raft_consensus.cc:1050] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd: attempting to promote NON_VOTER c3d26eafd51547acb780d4aac5ebe7cc to VOTER
I20240502 20:46:13.700839 9908 consensus_queue.cc:237] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } }
I20240502 20:46:13.701824 10432 tablet.cc:1722] T fd0c4a2a86604d39973dc7278c3a45a8 P ed47c023452a497192f35d2f617bf9bd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 20:46:13.702121 10432 tablet.cc:1620] T fd0c4a2a86604d39973dc7278c3a45a8 P ed47c023452a497192f35d2f617bf9bd: MemRowSet was empty: no flush needed.
I20240502 20:46:13.702714 4507 raft_consensus.cc:1261] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEARNER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:13.702888 6831 raft_consensus.cc:1261] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:13.703394 10217 consensus_queue.cc:1035] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:46:13.703898 4231 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: rebalancer_dynamic_table [id=28389bfda85a41a6b72a550afc0ef67d] alter complete (version 2)
I20240502 20:46:13.706156 10217 consensus_queue.cc:1035] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:46:13.706401 6967 raft_consensus.cc:1261] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:13.707973 10217 consensus_queue.cc:1035] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:46:13.708563 10677 raft_consensus.cc:2932] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } } }
I20240502 20:46:13.708891 6835 raft_consensus.cc:2932] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } } }
I20240502 20:46:13.711789 4230 catalog_manager.cc:5484] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd reported cstate change: config changed from index 2 to 3, c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:13.713065 4507 raft_consensus.cc:2932] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } } }
I20240502 20:46:13.714048 6967 raft_consensus.cc:2932] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } } }
I20240502 20:46:13.722106 10154 raft_consensus.cc:482] T b69500b0144a42f59383b8c5d38324b3 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:13.722227 10154 raft_consensus.cc:504] T b69500b0144a42f59383b8c5d38324b3 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:13.722555 10154 leader_election.cc:288] T b69500b0144a42f59383b8c5d38324b3 P 201d2b6485084e7dad2cbb722af94779 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
I20240502 20:46:13.722851 6967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b69500b0144a42f59383b8c5d38324b3" candidate_uuid: "201d2b6485084e7dad2cbb722af94779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47e5c56f13843018ac584914a8d4707" is_pre_election: true
I20240502 20:46:13.722867 6702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b69500b0144a42f59383b8c5d38324b3" candidate_uuid: "201d2b6485084e7dad2cbb722af94779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7845c397230d49358ba2fad9db53f97c" is_pre_election: true
I20240502 20:46:13.722982 6967 raft_consensus.cc:2454] T b69500b0144a42f59383b8c5d38324b3 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 201d2b6485084e7dad2cbb722af94779 in term 0.
I20240502 20:46:13.722991 6702 raft_consensus.cc:2454] T b69500b0144a42f59383b8c5d38324b3 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 201d2b6485084e7dad2cbb722af94779 in term 0.
I20240502 20:46:13.723240 7034 leader_election.cc:302] T b69500b0144a42f59383b8c5d38324b3 P 201d2b6485084e7dad2cbb722af94779 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201d2b6485084e7dad2cbb722af94779, 7845c397230d49358ba2fad9db53f97c; no voters:
I20240502 20:46:13.723451 10154 raft_consensus.cc:2781] T b69500b0144a42f59383b8c5d38324b3 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:13.723500 10154 raft_consensus.cc:482] T b69500b0144a42f59383b8c5d38324b3 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:13.723536 10154 raft_consensus.cc:3037] T b69500b0144a42f59383b8c5d38324b3 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:13.724362 10154 raft_consensus.cc:504] T b69500b0144a42f59383b8c5d38324b3 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:13.724720 10154 leader_election.cc:288] T b69500b0144a42f59383b8c5d38324b3 P 201d2b6485084e7dad2cbb722af94779 [CANDIDATE]: Term 1 election: Requested vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
I20240502 20:46:13.724967 6967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b69500b0144a42f59383b8c5d38324b3" candidate_uuid: "201d2b6485084e7dad2cbb722af94779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47e5c56f13843018ac584914a8d4707"
I20240502 20:46:13.725100 6967 raft_consensus.cc:3037] T b69500b0144a42f59383b8c5d38324b3 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:13.725064 6702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b69500b0144a42f59383b8c5d38324b3" candidate_uuid: "201d2b6485084e7dad2cbb722af94779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7845c397230d49358ba2fad9db53f97c"
I20240502 20:46:13.725205 6702 raft_consensus.cc:3037] T b69500b0144a42f59383b8c5d38324b3 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:13.725836 6967 raft_consensus.cc:2454] T b69500b0144a42f59383b8c5d38324b3 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 201d2b6485084e7dad2cbb722af94779 in term 1.
I20240502 20:46:13.726001 6702 raft_consensus.cc:2454] T b69500b0144a42f59383b8c5d38324b3 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 201d2b6485084e7dad2cbb722af94779 in term 1.
I20240502 20:46:13.726106 7035 leader_election.cc:302] T b69500b0144a42f59383b8c5d38324b3 P 201d2b6485084e7dad2cbb722af94779 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201d2b6485084e7dad2cbb722af94779, a47e5c56f13843018ac584914a8d4707; no voters:
I20240502 20:46:13.726318 10154 raft_consensus.cc:2781] T b69500b0144a42f59383b8c5d38324b3 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:13.726408 10154 raft_consensus.cc:686] T b69500b0144a42f59383b8c5d38324b3 P 201d2b6485084e7dad2cbb722af94779 [term 1 LEADER]: Becoming Leader. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Running, Role: LEADER
I20240502 20:46:13.726560 10154 consensus_queue.cc:237] T b69500b0144a42f59383b8c5d38324b3 P 201d2b6485084e7dad2cbb722af94779 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:13.728672 4230 catalog_manager.cc:5484] T b69500b0144a42f59383b8c5d38324b3 P 201d2b6485084e7dad2cbb722af94779 reported cstate change: term changed from 0 to 1, leader changed from <none> to 201d2b6485084e7dad2cbb722af94779 (127.30.32.72). New cstate: current_term: 1 leader_uuid: "201d2b6485084e7dad2cbb722af94779" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:13.744319 9929 raft_consensus.cc:482] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:13.744426 9929 raft_consensus.cc:504] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:13.744788 9929 leader_election.cc:288] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
I20240502 20:46:13.745134 6967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2508a5aa1c14b63971895953e149e17" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47e5c56f13843018ac584914a8d4707" is_pre_election: true
I20240502 20:46:13.745141 7099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2508a5aa1c14b63971895953e149e17" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201d2b6485084e7dad2cbb722af94779" is_pre_election: true
I20240502 20:46:13.745257 7099 raft_consensus.cc:2454] T b2508a5aa1c14b63971895953e149e17 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 0.
I20240502 20:46:13.745275 6967 raft_consensus.cc:2454] T b2508a5aa1c14b63971895953e149e17 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 0.
I20240502 20:46:13.745512 6768 leader_election.cc:302] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201d2b6485084e7dad2cbb722af94779, a4bf1df7fa6b4948be502c182bfbb833; no voters:
I20240502 20:46:13.745723 9929 raft_consensus.cc:2781] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:13.745770 9929 raft_consensus.cc:482] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:13.745811 9929 raft_consensus.cc:3037] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:13.746644 9929 raft_consensus.cc:504] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:13.746940 9929 leader_election.cc:288] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 election: Requested vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
I20240502 20:46:13.747233 6967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2508a5aa1c14b63971895953e149e17" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47e5c56f13843018ac584914a8d4707"
I20240502 20:46:13.747226 7099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2508a5aa1c14b63971895953e149e17" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201d2b6485084e7dad2cbb722af94779"
I20240502 20:46:13.747328 7099 raft_consensus.cc:3037] T b2508a5aa1c14b63971895953e149e17 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:13.747340 6967 raft_consensus.cc:3037] T b2508a5aa1c14b63971895953e149e17 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:13.748051 7099 raft_consensus.cc:2454] T b2508a5aa1c14b63971895953e149e17 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 1.
I20240502 20:46:13.748075 6967 raft_consensus.cc:2454] T b2508a5aa1c14b63971895953e149e17 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 1.
I20240502 20:46:13.748298 6768 leader_election.cc:302] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201d2b6485084e7dad2cbb722af94779, a4bf1df7fa6b4948be502c182bfbb833; no voters:
I20240502 20:46:13.748450 9929 raft_consensus.cc:2781] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:13.748618 9929 raft_consensus.cc:686] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEADER]: Becoming Leader. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Running, Role: LEADER
I20240502 20:46:13.748761 9929 consensus_queue.cc:237] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:13.750695 4230 catalog_manager.cc:5484] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70). New cstate: current_term: 1 leader_uuid: "a4bf1df7fa6b4948be502c182bfbb833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:13.777284 4769 raft_consensus.cc:1203] T a9afb75bfc114f59850eaea084a306f0 P ed47c023452a497192f35d2f617bf9bd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 20:46:13.778232 10788 ts_tablet_manager.cc:902] T 73efeba3d3094ef387e4efb03cff9d4e P a47e5c56f13843018ac584914a8d4707: Initiating tablet copy from peer 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:46:13.778767 10788 tablet_copy_client.cc:308] T 73efeba3d3094ef387e4efb03cff9d4e P a47e5c56f13843018ac584914a8d4707: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.67:42141
I20240502 20:46:13.779402 4658 tablet_copy_service.cc:140] P 923cfab926e7443f8ffd7b21ad0a9d91: Received BeginTabletCopySession request for tablet 73efeba3d3094ef387e4efb03cff9d4e from peer a47e5c56f13843018ac584914a8d4707 ({username='slave'} at 127.30.32.71:59213)
I20240502 20:46:13.779520 4658 tablet_copy_service.cc:161] P 923cfab926e7443f8ffd7b21ad0a9d91: Beginning new tablet copy session on tablet 73efeba3d3094ef387e4efb03cff9d4e from peer a47e5c56f13843018ac584914a8d4707 at {username='slave'} at 127.30.32.71:59213: session id = a47e5c56f13843018ac584914a8d4707-73efeba3d3094ef387e4efb03cff9d4e
I20240502 20:46:13.780679 4658 tablet_copy_source_session.cc:205] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:13.781673 10788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73efeba3d3094ef387e4efb03cff9d4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:13.784224 10788 tablet_copy_client.cc:783] T 73efeba3d3094ef387e4efb03cff9d4e P a47e5c56f13843018ac584914a8d4707: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:13.784401 10788 tablet_copy_client.cc:647] T 73efeba3d3094ef387e4efb03cff9d4e P a47e5c56f13843018ac584914a8d4707: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:13.785545 10788 tablet_copy_client.cc:523] T 73efeba3d3094ef387e4efb03cff9d4e P a47e5c56f13843018ac584914a8d4707: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:13.786725 10788 tablet_bootstrap.cc:492] T 73efeba3d3094ef387e4efb03cff9d4e P a47e5c56f13843018ac584914a8d4707: Bootstrap starting.
I20240502 20:46:13.808518 4227 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 20:46:13.814702 10757 tablet.cc:1722] T fd0c4a2a86604d39973dc7278c3a45a8 P 201d2b6485084e7dad2cbb722af94779: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 20:46:13.815551 10788 tablet_bootstrap.cc:492] T 73efeba3d3094ef387e4efb03cff9d4e P a47e5c56f13843018ac584914a8d4707: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:13.815867 10788 tablet_bootstrap.cc:492] T 73efeba3d3094ef387e4efb03cff9d4e P a47e5c56f13843018ac584914a8d4707: Bootstrap complete.
I20240502 20:46:13.816066 10788 ts_tablet_manager.cc:1367] T 73efeba3d3094ef387e4efb03cff9d4e P a47e5c56f13843018ac584914a8d4707: Time spent bootstrapping tablet: real 0.029s user 0.027s sys 0.000s
I20240502 20:46:13.816576 10788 raft_consensus.cc:348] T 73efeba3d3094ef387e4efb03cff9d4e P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } }
I20240502 20:46:13.816753 10788 raft_consensus.cc:729] T 73efeba3d3094ef387e4efb03cff9d4e P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Initialized, Role: LEARNER
I20240502 20:46:13.816881 10788 consensus_queue.cc:260] T 73efeba3d3094ef387e4efb03cff9d4e P a47e5c56f13843018ac584914a8d4707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } }
I20240502 20:46:13.816970 10432 tablet.cc:1722] T fd0c4a2a86604d39973dc7278c3a45a8 P ed47c023452a497192f35d2f617bf9bd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 20:46:13.818014 10788 ts_tablet_manager.cc:1398] T 73efeba3d3094ef387e4efb03cff9d4e P a47e5c56f13843018ac584914a8d4707: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:13.819329 4658 tablet_copy_service.cc:342] P 923cfab926e7443f8ffd7b21ad0a9d91: Request end of tablet copy session a47e5c56f13843018ac584914a8d4707-73efeba3d3094ef387e4efb03cff9d4e received from {username='slave'} at 127.30.32.71:59213
I20240502 20:46:13.819435 4658 tablet_copy_service.cc:433] P 923cfab926e7443f8ffd7b21ad0a9d91: ending tablet copy session a47e5c56f13843018ac584914a8d4707-73efeba3d3094ef387e4efb03cff9d4e on tablet 73efeba3d3094ef387e4efb03cff9d4e with peer a47e5c56f13843018ac584914a8d4707
I20240502 20:46:13.821328 10758 tablet.cc:1722] T fd0c4a2a86604d39973dc7278c3a45a8 P 7845c397230d49358ba2fad9db53f97c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 20:46:13.824355 4230 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: rebalancer_dynamic_table_new_name [id=28389bfda85a41a6b72a550afc0ef67d] alter complete (version 3)
I20240502 20:46:13.826455 4637 raft_consensus.cc:1203] T 93b7f7de1f8c49389c3d8c1e03c67f26 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 20:46:13.829197 4637 raft_consensus.cc:1203] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 20:46:13.845649 4769 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "78d87e2d8c0b4bbd8dd6d7f49cdb4287"
dest_uuid: "ed47c023452a497192f35d2f617bf9bd"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56294
I20240502 20:46:13.845808 4769 raft_consensus.cc:595] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Received request to transfer leadership
I20240502 20:46:13.880825 7099 consensus_queue.cc:237] T 51232fccd98c461fb3d9016081f52ba5 P 201d2b6485084e7dad2cbb722af94779 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } }
I20240502 20:46:13.882462 4507 raft_consensus.cc:1261] T 51232fccd98c461fb3d9016081f52ba5 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer 201d2b6485084e7dad2cbb722af94779: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:13.882561 6835 raft_consensus.cc:1261] T 51232fccd98c461fb3d9016081f52ba5 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Refusing update from remote peer 201d2b6485084e7dad2cbb722af94779: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:13.882829 10567 consensus_queue.cc:1035] T 51232fccd98c461fb3d9016081f52ba5 P 201d2b6485084e7dad2cbb722af94779 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 20:46:13.883008 7032 consensus_peers.cc:485] T 51232fccd98c461fb3d9016081f52ba5 P 201d2b6485084e7dad2cbb722af94779 -> Peer 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141): Couldn't send request to peer 923cfab926e7443f8ffd7b21ad0a9d91. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 51232fccd98c461fb3d9016081f52ba5. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:13.883361 10567 consensus_queue.cc:1035] T 51232fccd98c461fb3d9016081f52ba5 P 201d2b6485084e7dad2cbb722af94779 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:46:13.884511 10154 raft_consensus.cc:2932] T 51232fccd98c461fb3d9016081f52ba5 P 201d2b6485084e7dad2cbb722af94779 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } } }
I20240502 20:46:13.885282 4507 raft_consensus.cc:2932] T 51232fccd98c461fb3d9016081f52ba5 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } } }
I20240502 20:46:13.885458 6835 raft_consensus.cc:2932] T 51232fccd98c461fb3d9016081f52ba5 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } } }
I20240502 20:46:13.888484 4231 catalog_manager.cc:5484] T 51232fccd98c461fb3d9016081f52ba5 P 201d2b6485084e7dad2cbb722af94779 reported cstate change: config changed from index -1 to 2, NON_VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) added. New cstate: current_term: 1 leader_uuid: "201d2b6485084e7dad2cbb722af94779" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:13.926054 4227 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 20:46:13.926221 4227 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 20:46:13.928758 4227 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: Sending DeleteTablet for 3 replicas of tablet fd0c4a2a86604d39973dc7278c3a45a8
I20240502 20:46:13.929347 4750 tablet_service.cc:1514] Processing DeleteTablet for tablet fd0c4a2a86604d39973dc7278c3a45a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:13 UTC) from {username='slave'} at 127.0.0.1:38526
I20240502 20:46:13.929397 7079 tablet_service.cc:1514] Processing DeleteTablet for tablet fd0c4a2a86604d39973dc7278c3a45a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:13 UTC) from {username='slave'} at 127.0.0.1:59162
I20240502 20:46:13.929538 10638 tablet_replica.cc:330] T fd0c4a2a86604d39973dc7278c3a45a8 P ed47c023452a497192f35d2f617bf9bd: stopping tablet replica
I20240502 20:46:13.929682 10638 raft_consensus.cc:2229] T fd0c4a2a86604d39973dc7278c3a45a8 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:13.929719 10898 tablet_replica.cc:330] T fd0c4a2a86604d39973dc7278c3a45a8 P 201d2b6485084e7dad2cbb722af94779: stopping tablet replica
I20240502 20:46:13.929826 10638 raft_consensus.cc:2258] T fd0c4a2a86604d39973dc7278c3a45a8 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:13.929848 10898 raft_consensus.cc:2229] T fd0c4a2a86604d39973dc7278c3a45a8 P 201d2b6485084e7dad2cbb722af94779 [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:13.929986 6681 tablet_service.cc:1514] Processing DeleteTablet for tablet fd0c4a2a86604d39973dc7278c3a45a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:13 UTC) from {username='slave'} at 127.0.0.1:37506
I20240502 20:46:13.930047 10898 raft_consensus.cc:2258] T fd0c4a2a86604d39973dc7278c3a45a8 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:13.930459 10899 tablet_replica.cc:330] T fd0c4a2a86604d39973dc7278c3a45a8 P 7845c397230d49358ba2fad9db53f97c: stopping tablet replica
I20240502 20:46:13.930606 10899 raft_consensus.cc:2229] T fd0c4a2a86604d39973dc7278c3a45a8 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:13.930680 10898 ts_tablet_manager.cc:1875] T fd0c4a2a86604d39973dc7278c3a45a8 P 201d2b6485084e7dad2cbb722af94779: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:13.930680 10638 ts_tablet_manager.cc:1875] T fd0c4a2a86604d39973dc7278c3a45a8 P ed47c023452a497192f35d2f617bf9bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:13.930728 10899 raft_consensus.cc:2258] T fd0c4a2a86604d39973dc7278c3a45a8 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:13.931267 10899 ts_tablet_manager.cc:1875] T fd0c4a2a86604d39973dc7278c3a45a8 P 7845c397230d49358ba2fad9db53f97c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:13.933089 10638 ts_tablet_manager.cc:1888] T fd0c4a2a86604d39973dc7278c3a45a8 P ed47c023452a497192f35d2f617bf9bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:13.933090 10898 ts_tablet_manager.cc:1888] T fd0c4a2a86604d39973dc7278c3a45a8 P 201d2b6485084e7dad2cbb722af94779: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:13.933166 10898 log.cc:1196] T fd0c4a2a86604d39973dc7278c3a45a8 P 201d2b6485084e7dad2cbb722af94779: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/wal/wals/fd0c4a2a86604d39973dc7278c3a45a8
I20240502 20:46:13.933199 10899 ts_tablet_manager.cc:1888] T fd0c4a2a86604d39973dc7278c3a45a8 P 7845c397230d49358ba2fad9db53f97c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:13.933267 10638 log.cc:1196] T fd0c4a2a86604d39973dc7278c3a45a8 P ed47c023452a497192f35d2f617bf9bd: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal/wals/fd0c4a2a86604d39973dc7278c3a45a8
I20240502 20:46:13.933337 10899 log.cc:1196] T fd0c4a2a86604d39973dc7278c3a45a8 P 7845c397230d49358ba2fad9db53f97c: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/wal/wals/fd0c4a2a86604d39973dc7278c3a45a8
I20240502 20:46:13.933656 10638 ts_tablet_manager.cc:1909] T fd0c4a2a86604d39973dc7278c3a45a8 P ed47c023452a497192f35d2f617bf9bd: Deleting consensus metadata
I20240502 20:46:13.933658 10899 ts_tablet_manager.cc:1909] T fd0c4a2a86604d39973dc7278c3a45a8 P 7845c397230d49358ba2fad9db53f97c: Deleting consensus metadata
I20240502 20:46:13.934191 10898 ts_tablet_manager.cc:1909] T fd0c4a2a86604d39973dc7278c3a45a8 P 201d2b6485084e7dad2cbb722af94779: Deleting consensus metadata
I20240502 20:46:13.934396 4222 catalog_manager.cc:4830] TS ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193): tablet fd0c4a2a86604d39973dc7278c3a45a8 (table rebalancer_dynamic_table_new_name [id=28389bfda85a41a6b72a550afc0ef67d]) successfully deleted
I20240502 20:46:13.934576 4221 catalog_manager.cc:4830] TS 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): tablet fd0c4a2a86604d39973dc7278c3a45a8 (table rebalancer_dynamic_table_new_name [id=28389bfda85a41a6b72a550afc0ef67d]) successfully deleted
I20240502 20:46:13.935562 4219 catalog_manager.cc:4830] TS 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): tablet fd0c4a2a86604d39973dc7278c3a45a8 (table rebalancer_dynamic_table_new_name [id=28389bfda85a41a6b72a550afc0ef67d]) successfully deleted
I20240502 20:46:13.952423 4769 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "78d87e2d8c0b4bbd8dd6d7f49cdb4287"
dest_uuid: "ed47c023452a497192f35d2f617bf9bd"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56310
I20240502 20:46:13.952558 4769 raft_consensus.cc:595] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Received request to transfer leadership
I20240502 20:46:13.963160 9929 raft_consensus.cc:482] T 71537846373844cc9ef396b79af60db4 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:13.963290 9929 raft_consensus.cc:504] T 71537846373844cc9ef396b79af60db4 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:13.963678 9929 leader_election.cc:288] T 71537846373844cc9ef396b79af60db4 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
I20240502 20:46:13.964025 6703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71537846373844cc9ef396b79af60db4" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7845c397230d49358ba2fad9db53f97c" is_pre_election: true
I20240502 20:46:13.964082 7099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71537846373844cc9ef396b79af60db4" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201d2b6485084e7dad2cbb722af94779" is_pre_election: true
I20240502 20:46:13.964152 6703 raft_consensus.cc:2454] T 71537846373844cc9ef396b79af60db4 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 0.
I20240502 20:46:13.964233 7099 raft_consensus.cc:2454] T 71537846373844cc9ef396b79af60db4 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 0.
I20240502 20:46:13.964430 6770 leader_election.cc:302] T 71537846373844cc9ef396b79af60db4 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7845c397230d49358ba2fad9db53f97c, a4bf1df7fa6b4948be502c182bfbb833; no voters:
I20240502 20:46:13.964635 9929 raft_consensus.cc:2781] T 71537846373844cc9ef396b79af60db4 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:13.964709 9929 raft_consensus.cc:482] T 71537846373844cc9ef396b79af60db4 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:13.964762 9929 raft_consensus.cc:3037] T 71537846373844cc9ef396b79af60db4 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:13.965284 4374 raft_consensus.cc:1203] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 20:46:13.965940 9929 raft_consensus.cc:504] T 71537846373844cc9ef396b79af60db4 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:13.966326 9929 leader_election.cc:288] T 71537846373844cc9ef396b79af60db4 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 election: Requested vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
I20240502 20:46:13.966769 6703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71537846373844cc9ef396b79af60db4" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7845c397230d49358ba2fad9db53f97c"
I20240502 20:46:13.966887 6703 raft_consensus.cc:3037] T 71537846373844cc9ef396b79af60db4 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:13.966861 7099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71537846373844cc9ef396b79af60db4" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201d2b6485084e7dad2cbb722af94779"
I20240502 20:46:13.966974 7099 raft_consensus.cc:3037] T 71537846373844cc9ef396b79af60db4 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:13.967842 6703 raft_consensus.cc:2454] T 71537846373844cc9ef396b79af60db4 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 1.
I20240502 20:46:13.968045 7099 raft_consensus.cc:2454] T 71537846373844cc9ef396b79af60db4 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 1.
I20240502 20:46:13.968060 6770 leader_election.cc:302] T 71537846373844cc9ef396b79af60db4 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7845c397230d49358ba2fad9db53f97c, a4bf1df7fa6b4948be502c182bfbb833; no voters:
I20240502 20:46:13.968230 9929 raft_consensus.cc:2781] T 71537846373844cc9ef396b79af60db4 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:13.968334 9929 raft_consensus.cc:686] T 71537846373844cc9ef396b79af60db4 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEADER]: Becoming Leader. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Running, Role: LEADER
I20240502 20:46:13.968485 9929 consensus_queue.cc:237] T 71537846373844cc9ef396b79af60db4 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:13.970360 4235 catalog_manager.cc:5484] T 71537846373844cc9ef396b79af60db4 P a4bf1df7fa6b4948be502c182bfbb833 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70). New cstate: current_term: 1 leader_uuid: "a4bf1df7fa6b4948be502c182bfbb833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:14.019407 7099 raft_consensus.cc:1203] T d37730b109d04b6395f75c7552524877 P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 20:46:14.030859 4227 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41812:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 20:46:14.036226 7079 tablet_service.cc:1467] Processing CreateTablet for tablet 112cc778e5724fad8d4df9e9f88b4b9c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=39fbb150e9d84b52a68efaf1efbc90a9]), partition=
I20240502 20:46:14.036437 6945 tablet_service.cc:1467] Processing CreateTablet for tablet 112cc778e5724fad8d4df9e9f88b4b9c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=39fbb150e9d84b52a68efaf1efbc90a9]), partition=
I20240502 20:46:14.036522 7079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 112cc778e5724fad8d4df9e9f88b4b9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:14.036693 6945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 112cc778e5724fad8d4df9e9f88b4b9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:14.037425 6815 tablet_service.cc:1467] Processing CreateTablet for tablet 112cc778e5724fad8d4df9e9f88b4b9c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=39fbb150e9d84b52a68efaf1efbc90a9]), partition=
I20240502 20:46:14.037674 6815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 112cc778e5724fad8d4df9e9f88b4b9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:14.038791 10743 tablet_bootstrap.cc:492] T 112cc778e5724fad8d4df9e9f88b4b9c P a47e5c56f13843018ac584914a8d4707: Bootstrap starting.
I20240502 20:46:14.039685 10743 tablet_bootstrap.cc:654] T 112cc778e5724fad8d4df9e9f88b4b9c P a47e5c56f13843018ac584914a8d4707: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:14.039860 10742 tablet_bootstrap.cc:492] T 112cc778e5724fad8d4df9e9f88b4b9c P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap starting.
I20240502 20:46:14.040314 10653 tablet_bootstrap.cc:492] T 112cc778e5724fad8d4df9e9f88b4b9c P 201d2b6485084e7dad2cbb722af94779: Bootstrap starting.
I20240502 20:46:14.040813 10742 tablet_bootstrap.cc:654] T 112cc778e5724fad8d4df9e9f88b4b9c P a4bf1df7fa6b4948be502c182bfbb833: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:14.041266 10653 tablet_bootstrap.cc:654] T 112cc778e5724fad8d4df9e9f88b4b9c P 201d2b6485084e7dad2cbb722af94779: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:14.042325 10653 tablet_bootstrap.cc:492] T 112cc778e5724fad8d4df9e9f88b4b9c P 201d2b6485084e7dad2cbb722af94779: No bootstrap required, opened a new log
I20240502 20:46:14.042426 10653 ts_tablet_manager.cc:1367] T 112cc778e5724fad8d4df9e9f88b4b9c P 201d2b6485084e7dad2cbb722af94779: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:14.042618 10743 tablet_bootstrap.cc:492] T 112cc778e5724fad8d4df9e9f88b4b9c P a47e5c56f13843018ac584914a8d4707: No bootstrap required, opened a new log
I20240502 20:46:14.042707 10743 ts_tablet_manager.cc:1367] T 112cc778e5724fad8d4df9e9f88b4b9c P a47e5c56f13843018ac584914a8d4707: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 20:46:14.042901 10653 raft_consensus.cc:348] T 112cc778e5724fad8d4df9e9f88b4b9c P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:14.042996 10653 raft_consensus.cc:374] T 112cc778e5724fad8d4df9e9f88b4b9c P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:14.043018 10653 raft_consensus.cc:729] T 112cc778e5724fad8d4df9e9f88b4b9c P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Initialized, Role: FOLLOWER
I20240502 20:46:14.043145 10653 consensus_queue.cc:260] T 112cc778e5724fad8d4df9e9f88b4b9c P 201d2b6485084e7dad2cbb722af94779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:14.043161 10743 raft_consensus.cc:348] T 112cc778e5724fad8d4df9e9f88b4b9c P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:14.043265 10743 raft_consensus.cc:374] T 112cc778e5724fad8d4df9e9f88b4b9c P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:14.043303 10743 raft_consensus.cc:729] T 112cc778e5724fad8d4df9e9f88b4b9c P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Initialized, Role: FOLLOWER
I20240502 20:46:14.043431 10653 ts_tablet_manager.cc:1398] T 112cc778e5724fad8d4df9e9f88b4b9c P 201d2b6485084e7dad2cbb722af94779: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:14.043426 10743 consensus_queue.cc:260] T 112cc778e5724fad8d4df9e9f88b4b9c P a47e5c56f13843018ac584914a8d4707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:14.043689 10743 ts_tablet_manager.cc:1398] T 112cc778e5724fad8d4df9e9f88b4b9c P a47e5c56f13843018ac584914a8d4707: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:14.044701 10742 tablet_bootstrap.cc:492] T 112cc778e5724fad8d4df9e9f88b4b9c P a4bf1df7fa6b4948be502c182bfbb833: No bootstrap required, opened a new log
I20240502 20:46:14.044799 10742 ts_tablet_manager.cc:1367] T 112cc778e5724fad8d4df9e9f88b4b9c P a4bf1df7fa6b4948be502c182bfbb833: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 20:46:14.045279 10742 raft_consensus.cc:348] T 112cc778e5724fad8d4df9e9f88b4b9c P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:14.045382 10742 raft_consensus.cc:374] T 112cc778e5724fad8d4df9e9f88b4b9c P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:14.045423 10742 raft_consensus.cc:729] T 112cc778e5724fad8d4df9e9f88b4b9c P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Initialized, Role: FOLLOWER
I20240502 20:46:14.045538 10742 consensus_queue.cc:260] T 112cc778e5724fad8d4df9e9f88b4b9c P a4bf1df7fa6b4948be502c182bfbb833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:14.045821 10742 ts_tablet_manager.cc:1398] T 112cc778e5724fad8d4df9e9f88b4b9c P a4bf1df7fa6b4948be502c182bfbb833: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:14.104043 9105 raft_consensus.cc:1050] T 3d938d01d5154590a9262acd54094ad2 P 7845c397230d49358ba2fad9db53f97c: attempting to promote NON_VOTER 923cfab926e7443f8ffd7b21ad0a9d91 to VOTER
I20240502 20:46:14.104494 9105 consensus_queue.cc:237] T 3d938d01d5154590a9262acd54094ad2 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } }
I20240502 20:46:14.105906 4637 raft_consensus.cc:1261] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEARNER]: Refusing update from remote peer 7845c397230d49358ba2fad9db53f97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:14.105906 6967 raft_consensus.cc:1261] T 3d938d01d5154590a9262acd54094ad2 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Refusing update from remote peer 7845c397230d49358ba2fad9db53f97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:14.105912 7099 raft_consensus.cc:1261] T 3d938d01d5154590a9262acd54094ad2 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Refusing update from remote peer 7845c397230d49358ba2fad9db53f97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:14.106338 9105 consensus_queue.cc:1035] T 3d938d01d5154590a9262acd54094ad2 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:46:14.106550 10922 consensus_queue.cc:1035] T 3d938d01d5154590a9262acd54094ad2 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:46:14.106734 10270 consensus_queue.cc:1035] T 3d938d01d5154590a9262acd54094ad2 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:46:14.109170 10922 raft_consensus.cc:2932] T 3d938d01d5154590a9262acd54094ad2 P 7845c397230d49358ba2fad9db53f97c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:14.109797 4637 raft_consensus.cc:2932] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:14.111197 6967 raft_consensus.cc:2932] T 3d938d01d5154590a9262acd54094ad2 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:14.111735 7099 raft_consensus.cc:2932] T 3d938d01d5154590a9262acd54094ad2 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:14.113032 4227 catalog_manager.cc:5484] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index 2 to 3, 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7845c397230d49358ba2fad9db53f97c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:14.117187 6703 consensus_queue.cc:237] T 3d938d01d5154590a9262acd54094ad2 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } }
I20240502 20:46:14.118341 7099 raft_consensus.cc:1261] T 3d938d01d5154590a9262acd54094ad2 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Refusing update from remote peer 7845c397230d49358ba2fad9db53f97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:14.118451 4637 raft_consensus.cc:1261] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer 7845c397230d49358ba2fad9db53f97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:14.118723 9105 consensus_queue.cc:1035] T 3d938d01d5154590a9262acd54094ad2 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:14.119071 10504 consensus_queue.cc:1035] T 3d938d01d5154590a9262acd54094ad2 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:14.120608 10504 raft_consensus.cc:2932] T 3d938d01d5154590a9262acd54094ad2 P 7845c397230d49358ba2fad9db53f97c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:14.121331 7099 raft_consensus.cc:2932] T 3d938d01d5154590a9262acd54094ad2 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:14.122130 4221 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3d938d01d5154590a9262acd54094ad2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:14.122783 4637 raft_consensus.cc:2932] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:14.123302 4235 catalog_manager.cc:5484] T 3d938d01d5154590a9262acd54094ad2 P 7845c397230d49358ba2fad9db53f97c reported cstate change: config changed from index 3 to 4, VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) evicted. New cstate: current_term: 1 leader_uuid: "7845c397230d49358ba2fad9db53f97c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:14.126096 6945 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d938d01d5154590a9262acd54094ad2 with delete_type TABLET_DATA_TOMBSTONED (TS a47e5c56f13843018ac584914a8d4707 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45156
I20240502 20:46:14.126459 10926 tablet_replica.cc:330] T 3d938d01d5154590a9262acd54094ad2 P a47e5c56f13843018ac584914a8d4707: stopping tablet replica
I20240502 20:46:14.126628 10926 raft_consensus.cc:2229] T 3d938d01d5154590a9262acd54094ad2 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:14.126797 10926 raft_consensus.cc:2258] T 3d938d01d5154590a9262acd54094ad2 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:14.127542 10926 ts_tablet_manager.cc:1875] T 3d938d01d5154590a9262acd54094ad2 P a47e5c56f13843018ac584914a8d4707: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:14.129416 10926 ts_tablet_manager.cc:1888] T 3d938d01d5154590a9262acd54094ad2 P a47e5c56f13843018ac584914a8d4707: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 20:46:14.129488 10926 log.cc:1196] T 3d938d01d5154590a9262acd54094ad2 P a47e5c56f13843018ac584914a8d4707: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/wal/wals/3d938d01d5154590a9262acd54094ad2
I20240502 20:46:14.130012 4222 catalog_manager.cc:4830] TS a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): tablet 3d938d01d5154590a9262acd54094ad2 (table rebalancer_extra_table_4 [id=1bde7089cdee45b68bdc73a007b74174]) successfully deleted
I20240502 20:46:14.134822 10776 consensus_queue.cc:1035] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:14.138597 10776 consensus_queue.cc:1035] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:14.142341 9125 raft_consensus.cc:979] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd: : Instructing follower a4bf1df7fa6b4948be502c182bfbb833 to start an election
I20240502 20:46:14.142454 9125 raft_consensus.cc:1067] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Signalling peer a4bf1df7fa6b4948be502c182bfbb833 to start an election
I20240502 20:46:14.142917 6835 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "78d87e2d8c0b4bbd8dd6d7f49cdb4287"
dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833"
from {username='slave'} at 127.30.32.68:59919
I20240502 20:46:14.143079 6835 raft_consensus.cc:482] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:46:14.143144 6835 raft_consensus.cc:3037] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:14.144045 6835 raft_consensus.cc:504] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a4bf1df7fa6b4948be502c182bfbb833 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } }
I20240502 20:46:14.144435 6835 leader_election.cc:288] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 election: Requested vote from peers ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097)
I20240502 20:46:14.144829 4769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78d87e2d8c0b4bbd8dd6d7f49cdb4287" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ed47c023452a497192f35d2f617bf9bd"
I20240502 20:46:14.144850 4507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78d87e2d8c0b4bbd8dd6d7f49cdb4287" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc"
I20240502 20:46:14.144968 4769 raft_consensus.cc:3032] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:46:14.144934 6967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78d87e2d8c0b4bbd8dd6d7f49cdb4287" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "a47e5c56f13843018ac584914a8d4707"
I20240502 20:46:14.145018 4507 raft_consensus.cc:3037] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:14.145048 4769 raft_consensus.cc:729] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Running, Role: LEADER
I20240502 20:46:14.145084 6967 raft_consensus.cc:3037] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:14.145210 4769 consensus_queue.cc:260] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } }
I20240502 20:46:14.145377 4769 raft_consensus.cc:3037] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:14.146061 6967 raft_consensus.cc:2454] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a47e5c56f13843018ac584914a8d4707 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 2.
I20240502 20:46:14.146061 4507 raft_consensus.cc:2454] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P c3d26eafd51547acb780d4aac5ebe7cc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 2.
I20240502 20:46:14.146175 4769 raft_consensus.cc:2454] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 2.
I20240502 20:46:14.146504 6771 leader_election.cc:302] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: a4bf1df7fa6b4948be502c182bfbb833, c3d26eafd51547acb780d4aac5ebe7cc, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:46:14.146701 10776 raft_consensus.cc:2781] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a4bf1df7fa6b4948be502c182bfbb833 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:46:14.146812 10776 raft_consensus.cc:686] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a4bf1df7fa6b4948be502c182bfbb833 [term 2 LEADER]: Becoming Leader. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Running, Role: LEADER
I20240502 20:46:14.146968 10776 consensus_queue.cc:237] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } }
I20240502 20:46:14.149200 4231 catalog_manager.cc:5484] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a4bf1df7fa6b4948be502c182bfbb833 reported cstate change: term changed from 1 to 2, leader changed from ed47c023452a497192f35d2f617bf9bd (127.30.32.68) to a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70). New cstate: current_term: 2 leader_uuid: "a4bf1df7fa6b4948be502c182bfbb833" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:14.153514 10154 raft_consensus.cc:482] T 112cc778e5724fad8d4df9e9f88b4b9c P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:14.153604 10154 raft_consensus.cc:504] T 112cc778e5724fad8d4df9e9f88b4b9c P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:14.154016 10154 leader_election.cc:288] T 112cc778e5724fad8d4df9e9f88b4b9c P 201d2b6485084e7dad2cbb722af94779 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867)
I20240502 20:46:14.154341 6967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "112cc778e5724fad8d4df9e9f88b4b9c" candidate_uuid: "201d2b6485084e7dad2cbb722af94779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47e5c56f13843018ac584914a8d4707" is_pre_election: true
I20240502 20:46:14.154476 6967 raft_consensus.cc:2454] T 112cc778e5724fad8d4df9e9f88b4b9c P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 201d2b6485084e7dad2cbb722af94779 in term 0.
I20240502 20:46:14.154736 7035 leader_election.cc:302] T 112cc778e5724fad8d4df9e9f88b4b9c P 201d2b6485084e7dad2cbb722af94779 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201d2b6485084e7dad2cbb722af94779, a47e5c56f13843018ac584914a8d4707; no voters:
I20240502 20:46:14.154845 6835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "112cc778e5724fad8d4df9e9f88b4b9c" candidate_uuid: "201d2b6485084e7dad2cbb722af94779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833" is_pre_election: true
I20240502 20:46:14.154908 10154 raft_consensus.cc:2781] T 112cc778e5724fad8d4df9e9f88b4b9c P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:14.154973 10154 raft_consensus.cc:482] T 112cc778e5724fad8d4df9e9f88b4b9c P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:14.154973 6835 raft_consensus.cc:2454] T 112cc778e5724fad8d4df9e9f88b4b9c P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 201d2b6485084e7dad2cbb722af94779 in term 0.
I20240502 20:46:14.155022 10154 raft_consensus.cc:3037] T 112cc778e5724fad8d4df9e9f88b4b9c P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:14.156005 10154 raft_consensus.cc:504] T 112cc778e5724fad8d4df9e9f88b4b9c P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:14.156376 10154 leader_election.cc:288] T 112cc778e5724fad8d4df9e9f88b4b9c P 201d2b6485084e7dad2cbb722af94779 [CANDIDATE]: Term 1 election: Requested vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867)
I20240502 20:46:14.156776 6967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "112cc778e5724fad8d4df9e9f88b4b9c" candidate_uuid: "201d2b6485084e7dad2cbb722af94779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47e5c56f13843018ac584914a8d4707"
I20240502 20:46:14.156821 6835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "112cc778e5724fad8d4df9e9f88b4b9c" candidate_uuid: "201d2b6485084e7dad2cbb722af94779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833"
I20240502 20:46:14.156888 6967 raft_consensus.cc:3037] T 112cc778e5724fad8d4df9e9f88b4b9c P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:14.156924 6835 raft_consensus.cc:3037] T 112cc778e5724fad8d4df9e9f88b4b9c P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:14.157835 6967 raft_consensus.cc:2454] T 112cc778e5724fad8d4df9e9f88b4b9c P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 201d2b6485084e7dad2cbb722af94779 in term 1.
I20240502 20:46:14.157907 6835 raft_consensus.cc:2454] T 112cc778e5724fad8d4df9e9f88b4b9c P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 201d2b6485084e7dad2cbb722af94779 in term 1.
I20240502 20:46:14.158133 7035 leader_election.cc:302] T 112cc778e5724fad8d4df9e9f88b4b9c P 201d2b6485084e7dad2cbb722af94779 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201d2b6485084e7dad2cbb722af94779, a47e5c56f13843018ac584914a8d4707; no voters:
I20240502 20:46:14.158440 10154 raft_consensus.cc:2781] T 112cc778e5724fad8d4df9e9f88b4b9c P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:14.158535 10154 raft_consensus.cc:686] T 112cc778e5724fad8d4df9e9f88b4b9c P 201d2b6485084e7dad2cbb722af94779 [term 1 LEADER]: Becoming Leader. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Running, Role: LEADER
I20240502 20:46:14.158664 10154 consensus_queue.cc:237] T 112cc778e5724fad8d4df9e9f88b4b9c P 201d2b6485084e7dad2cbb722af94779 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:14.160466 4231 catalog_manager.cc:5484] T 112cc778e5724fad8d4df9e9f88b4b9c P 201d2b6485084e7dad2cbb722af94779 reported cstate change: term changed from 0 to 1, leader changed from <none> to 201d2b6485084e7dad2cbb722af94779 (127.30.32.72). New cstate: current_term: 1 leader_uuid: "201d2b6485084e7dad2cbb722af94779" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:14.225533 10154 consensus_queue.cc:1035] T b69500b0144a42f59383b8c5d38324b3 P 201d2b6485084e7dad2cbb722af94779 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:14.226462 9105 raft_consensus.cc:1050] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c: attempting to promote NON_VOTER ed47c023452a497192f35d2f617bf9bd to VOTER
I20240502 20:46:14.226855 9105 consensus_queue.cc:237] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } }
I20240502 20:46:14.228724 6835 raft_consensus.cc:1261] T a9afb75bfc114f59850eaea084a306f0 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Refusing update from remote peer 7845c397230d49358ba2fad9db53f97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:14.228767 4769 raft_consensus.cc:1261] T a9afb75bfc114f59850eaea084a306f0 P ed47c023452a497192f35d2f617bf9bd [term 1 LEARNER]: Refusing update from remote peer 7845c397230d49358ba2fad9db53f97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:14.229099 10922 consensus_queue.cc:1035] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:46:14.229132 6967 raft_consensus.cc:1261] T a9afb75bfc114f59850eaea084a306f0 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Refusing update from remote peer 7845c397230d49358ba2fad9db53f97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:14.229782 10504 consensus_queue.cc:1035] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:46:14.230113 10504 consensus_queue.cc:1035] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:46:14.233232 4637 consensus_queue.cc:237] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } }
W20240502 20:46:14.235061 4575 consensus_peers.cc:485] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 28da7347f982468281570ffa7503a854. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:14.235569 4507 raft_consensus.cc:1261] T 28da7347f982468281570ffa7503a854 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:14.235960 9041 consensus_queue.cc:1035] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:14.236363 6835 raft_consensus.cc:1261] T 28da7347f982468281570ffa7503a854 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:14.238729 10154 consensus_queue.cc:1035] T b69500b0144a42f59383b8c5d38324b3 P 201d2b6485084e7dad2cbb722af94779 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:14.240159 9041 consensus_queue.cc:1035] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:14.240439 10922 raft_consensus.cc:2932] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ed47c023452a497192f35d2f617bf9bd (127.30.32.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } } }
I20240502 20:46:14.241833 6957 raft_consensus.cc:2932] T a9afb75bfc114f59850eaea084a306f0 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ed47c023452a497192f35d2f617bf9bd (127.30.32.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } } }
I20240502 20:46:14.244181 4230 catalog_manager.cc:5484] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c reported cstate change: config changed from index 2 to 3, ed47c023452a497192f35d2f617bf9bd (127.30.32.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7845c397230d49358ba2fad9db53f97c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:14.244998 6831 raft_consensus.cc:2932] T a9afb75bfc114f59850eaea084a306f0 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ed47c023452a497192f35d2f617bf9bd (127.30.32.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } } }
I20240502 20:46:14.247866 4768 raft_consensus.cc:1203] T 9258667ccc3e4ef8b26fe75531dd4591 P ed47c023452a497192f35d2f617bf9bd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 20:46:14.249428 10994 raft_consensus.cc:2932] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } } }
I20240502 20:46:14.251189 4507 raft_consensus.cc:2932] T 28da7347f982468281570ffa7503a854 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } } }
I20240502 20:46:14.251750 4768 raft_consensus.cc:2932] T a9afb75bfc114f59850eaea084a306f0 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ed47c023452a497192f35d2f617bf9bd (127.30.32.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } } }
I20240502 20:46:14.253319 4235 catalog_manager.cc:5484] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index 6 to 7, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:14.259271 6835 raft_consensus.cc:2932] T 28da7347f982468281570ffa7503a854 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } } }
I20240502 20:46:14.290846 4235 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 20:46:14.295234 6835 raft_consensus.cc:1261] T 112cc778e5724fad8d4df9e9f88b4b9c P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Refusing update from remote peer 201d2b6485084e7dad2cbb722af94779: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:14.295256 6967 raft_consensus.cc:1261] T 112cc778e5724fad8d4df9e9f88b4b9c P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Refusing update from remote peer 201d2b6485084e7dad2cbb722af94779: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:14.295677 10567 consensus_queue.cc:1035] T 112cc778e5724fad8d4df9e9f88b4b9c P 201d2b6485084e7dad2cbb722af94779 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:14.296023 8460 consensus_queue.cc:1035] T 112cc778e5724fad8d4df9e9f88b4b9c P 201d2b6485084e7dad2cbb722af94779 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:14.298520 4235 catalog_manager.cc:5576] TS a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841) does not have the latest schema for tablet 112cc778e5724fad8d4df9e9f88b4b9c (table rebalancer_dynamic_table [id=39fbb150e9d84b52a68efaf1efbc90a9]). Expected version 1 got 0
I20240502 20:46:14.299722 10757 tablet.cc:1722] T 112cc778e5724fad8d4df9e9f88b4b9c P 201d2b6485084e7dad2cbb722af94779: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 20:46:14.300071 10757 tablet.cc:1620] T 112cc778e5724fad8d4df9e9f88b4b9c P 201d2b6485084e7dad2cbb722af94779: MemRowSet was empty: no flush needed.
I20240502 20:46:14.300437 4230 catalog_manager.cc:5576] TS a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867) does not have the latest schema for tablet 112cc778e5724fad8d4df9e9f88b4b9c (table rebalancer_dynamic_table [id=39fbb150e9d84b52a68efaf1efbc90a9]). Expected version 1 got 0
I20240502 20:46:14.302217 11030 tablet.cc:1722] T 112cc778e5724fad8d4df9e9f88b4b9c P a4bf1df7fa6b4948be502c182bfbb833: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 20:46:14.302814 11025 tablet.cc:1722] T 112cc778e5724fad8d4df9e9f88b4b9c P a47e5c56f13843018ac584914a8d4707: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 20:46:14.303126 11025 tablet.cc:1620] T 112cc778e5724fad8d4df9e9f88b4b9c P a47e5c56f13843018ac584914a8d4707: MemRowSet was empty: no flush needed.
I20240502 20:46:14.303527 11030 tablet.cc:1620] T 112cc778e5724fad8d4df9e9f88b4b9c P a4bf1df7fa6b4948be502c182bfbb833: MemRowSet was empty: no flush needed.
I20240502 20:46:14.303825 4231 catalog_manager.cc:5576] TS a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867) does not have the latest schema for tablet 112cc778e5724fad8d4df9e9f88b4b9c (table rebalancer_dynamic_table [id=39fbb150e9d84b52a68efaf1efbc90a9]). Expected version 1 got 0
I20240502 20:46:14.304563 4230 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: rebalancer_dynamic_table [id=39fbb150e9d84b52a68efaf1efbc90a9] alter complete (version 1)
I20240502 20:46:14.305527 4235 catalog_manager.cc:5576] TS a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841) does not have the latest schema for tablet 112cc778e5724fad8d4df9e9f88b4b9c (table rebalancer_dynamic_table [id=39fbb150e9d84b52a68efaf1efbc90a9]). Expected version 1 got 0
I20240502 20:46:14.311975 6967 raft_consensus.cc:1203] T 73efeba3d3094ef387e4efb03cff9d4e P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 20:46:14.327363 6703 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a9afb75bfc114f59850eaea084a306f0"
dest_uuid: "7845c397230d49358ba2fad9db53f97c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52152
I20240502 20:46:14.327572 6703 raft_consensus.cc:595] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c [term 1 LEADER]: Received request to transfer leadership
I20240502 20:46:14.363746 10270 raft_consensus.cc:1050] T a135d04d8c3248bc954a07e5b6396fde P 7845c397230d49358ba2fad9db53f97c: attempting to promote NON_VOTER 923cfab926e7443f8ffd7b21ad0a9d91 to VOTER
I20240502 20:46:14.364156 10270 consensus_queue.cc:237] T a135d04d8c3248bc954a07e5b6396fde P 7845c397230d49358ba2fad9db53f97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } }
I20240502 20:46:14.365468 4637 raft_consensus.cc:1261] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEARNER]: Refusing update from remote peer 7845c397230d49358ba2fad9db53f97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:14.365511 4374 raft_consensus.cc:1261] T a135d04d8c3248bc954a07e5b6396fde P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer 7845c397230d49358ba2fad9db53f97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:14.365569 7099 raft_consensus.cc:1261] T a135d04d8c3248bc954a07e5b6396fde P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Refusing update from remote peer 7845c397230d49358ba2fad9db53f97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:14.365931 10270 consensus_queue.cc:1035] T a135d04d8c3248bc954a07e5b6396fde P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:46:14.366108 10922 consensus_queue.cc:1035] T a135d04d8c3248bc954a07e5b6396fde P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:46:14.366333 9105 consensus_queue.cc:1035] T a135d04d8c3248bc954a07e5b6396fde P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:46:14.368250 10270 raft_consensus.cc:2932] T a135d04d8c3248bc954a07e5b6396fde P 7845c397230d49358ba2fad9db53f97c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:14.369038 7099 raft_consensus.cc:2932] T a135d04d8c3248bc954a07e5b6396fde P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:14.370620 4374 raft_consensus.cc:2932] T a135d04d8c3248bc954a07e5b6396fde P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:14.370679 4637 raft_consensus.cc:2932] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:14.371964 4235 catalog_manager.cc:5484] T a135d04d8c3248bc954a07e5b6396fde P 7845c397230d49358ba2fad9db53f97c reported cstate change: config changed from index 2 to 3, 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7845c397230d49358ba2fad9db53f97c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:14.377841 6703 consensus_queue.cc:237] T a135d04d8c3248bc954a07e5b6396fde P 7845c397230d49358ba2fad9db53f97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } }
I20240502 20:46:14.378934 7099 raft_consensus.cc:1261] T a135d04d8c3248bc954a07e5b6396fde P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Refusing update from remote peer 7845c397230d49358ba2fad9db53f97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:14.378948 4637 raft_consensus.cc:1261] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer 7845c397230d49358ba2fad9db53f97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:14.379305 10504 consensus_queue.cc:1035] T a135d04d8c3248bc954a07e5b6396fde P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:14.379494 10270 consensus_queue.cc:1035] T a135d04d8c3248bc954a07e5b6396fde P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:14.381084 10922 raft_consensus.cc:2932] T a135d04d8c3248bc954a07e5b6396fde P 7845c397230d49358ba2fad9db53f97c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:14.382330 7099 raft_consensus.cc:2932] T a135d04d8c3248bc954a07e5b6396fde P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:14.382470 4221 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a135d04d8c3248bc954a07e5b6396fde with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:14.382752 4637 raft_consensus.cc:2932] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:14.384549 4231 catalog_manager.cc:5484] T a135d04d8c3248bc954a07e5b6396fde P 7845c397230d49358ba2fad9db53f97c reported cstate change: config changed from index 3 to 4, VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) evicted. New cstate: current_term: 1 leader_uuid: "7845c397230d49358ba2fad9db53f97c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:14.390455 4354 tablet_service.cc:1514] Processing DeleteTablet for tablet a135d04d8c3248bc954a07e5b6396fde with delete_type TABLET_DATA_TOMBSTONED (TS c92e6e46782849479c8a0c6d82276fb2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38306
I20240502 20:46:14.396791 11061 tablet_replica.cc:330] T a135d04d8c3248bc954a07e5b6396fde P c92e6e46782849479c8a0c6d82276fb2: stopping tablet replica
I20240502 20:46:14.396942 11061 raft_consensus.cc:2229] T a135d04d8c3248bc954a07e5b6396fde P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:14.397107 11061 raft_consensus.cc:2258] T a135d04d8c3248bc954a07e5b6396fde P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:14.397689 11061 ts_tablet_manager.cc:1875] T a135d04d8c3248bc954a07e5b6396fde P c92e6e46782849479c8a0c6d82276fb2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:14.399250 11062 ts_tablet_manager.cc:902] T 51232fccd98c461fb3d9016081f52ba5 P 923cfab926e7443f8ffd7b21ad0a9d91: Initiating tablet copy from peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
I20240502 20:46:14.399796 11062 tablet_copy_client.cc:308] T 51232fccd98c461fb3d9016081f52ba5 P 923cfab926e7443f8ffd7b21ad0a9d91: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.72:44643
I20240502 20:46:14.399950 11061 ts_tablet_manager.cc:1888] T a135d04d8c3248bc954a07e5b6396fde P c92e6e46782849479c8a0c6d82276fb2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 20:46:14.400025 11061 log.cc:1196] T a135d04d8c3248bc954a07e5b6396fde P c92e6e46782849479c8a0c6d82276fb2: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-0/wal/wals/a135d04d8c3248bc954a07e5b6396fde
I20240502 20:46:14.400439 7119 tablet_copy_service.cc:140] P 201d2b6485084e7dad2cbb722af94779: Received BeginTabletCopySession request for tablet 51232fccd98c461fb3d9016081f52ba5 from peer 923cfab926e7443f8ffd7b21ad0a9d91 ({username='slave'} at 127.30.32.67:34803)
I20240502 20:46:14.400568 7119 tablet_copy_service.cc:161] P 201d2b6485084e7dad2cbb722af94779: Beginning new tablet copy session on tablet 51232fccd98c461fb3d9016081f52ba5 from peer 923cfab926e7443f8ffd7b21ad0a9d91 at {username='slave'} at 127.30.32.67:34803: session id = 923cfab926e7443f8ffd7b21ad0a9d91-51232fccd98c461fb3d9016081f52ba5
I20240502 20:46:14.400660 4222 catalog_manager.cc:4830] TS c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827): tablet a135d04d8c3248bc954a07e5b6396fde (table rebalancer_extra_table_14 [id=319cff57d33e47f69faa288db2be515b]) successfully deleted
I20240502 20:46:14.401790 7119 tablet_copy_source_session.cc:205] T 51232fccd98c461fb3d9016081f52ba5 P 201d2b6485084e7dad2cbb722af94779: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:14.403486 11062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51232fccd98c461fb3d9016081f52ba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:14.405433 11062 tablet_copy_client.cc:783] T 51232fccd98c461fb3d9016081f52ba5 P 923cfab926e7443f8ffd7b21ad0a9d91: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:14.405584 11062 tablet_copy_client.cc:647] T 51232fccd98c461fb3d9016081f52ba5 P 923cfab926e7443f8ffd7b21ad0a9d91: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:14.406771 11062 tablet_copy_client.cc:523] T 51232fccd98c461fb3d9016081f52ba5 P 923cfab926e7443f8ffd7b21ad0a9d91: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:14.407740 11062 tablet_bootstrap.cc:492] T 51232fccd98c461fb3d9016081f52ba5 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:46:14.408864 4230 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 20:46:14.411229 11062 tablet_bootstrap.cc:492] T 51232fccd98c461fb3d9016081f52ba5 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:14.411490 11062 tablet_bootstrap.cc:492] T 51232fccd98c461fb3d9016081f52ba5 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap complete.
I20240502 20:46:14.411638 11062 ts_tablet_manager.cc:1367] T 51232fccd98c461fb3d9016081f52ba5 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 20:46:14.412380 11062 raft_consensus.cc:348] T 51232fccd98c461fb3d9016081f52ba5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } }
I20240502 20:46:14.412500 11062 raft_consensus.cc:729] T 51232fccd98c461fb3d9016081f52ba5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: LEARNER
I20240502 20:46:14.412617 11062 consensus_queue.cc:260] T 51232fccd98c461fb3d9016081f52ba5 P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } }
I20240502 20:46:14.412961 11062 ts_tablet_manager.cc:1398] T 51232fccd98c461fb3d9016081f52ba5 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 20:46:14.415321 7119 tablet_copy_service.cc:342] P 201d2b6485084e7dad2cbb722af94779: Request end of tablet copy session 923cfab926e7443f8ffd7b21ad0a9d91-51232fccd98c461fb3d9016081f52ba5 received from {username='slave'} at 127.30.32.67:34803
I20240502 20:46:14.415412 7119 tablet_copy_service.cc:433] P 201d2b6485084e7dad2cbb722af94779: ending tablet copy session 923cfab926e7443f8ffd7b21ad0a9d91-51232fccd98c461fb3d9016081f52ba5 on tablet 51232fccd98c461fb3d9016081f52ba5 with peer 923cfab926e7443f8ffd7b21ad0a9d91
I20240502 20:46:14.415704 10757 tablet.cc:1722] T 112cc778e5724fad8d4df9e9f88b4b9c P 201d2b6485084e7dad2cbb722af94779: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 20:46:14.415988 10757 tablet.cc:1620] T 112cc778e5724fad8d4df9e9f88b4b9c P 201d2b6485084e7dad2cbb722af94779: MemRowSet was empty: no flush needed.
I20240502 20:46:14.416253 11025 tablet.cc:1722] T 112cc778e5724fad8d4df9e9f88b4b9c P a47e5c56f13843018ac584914a8d4707: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 20:46:14.416517 11025 tablet.cc:1620] T 112cc778e5724fad8d4df9e9f88b4b9c P a47e5c56f13843018ac584914a8d4707: MemRowSet was empty: no flush needed.
I20240502 20:46:14.420747 8185 raft_consensus.cc:1050] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2: attempting to promote NON_VOTER 923cfab926e7443f8ffd7b21ad0a9d91 to VOTER
I20240502 20:46:14.421231 8185 consensus_queue.cc:237] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } }
I20240502 20:46:14.421921 4231 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: rebalancer_dynamic_table [id=39fbb150e9d84b52a68efaf1efbc90a9] alter complete (version 2)
I20240502 20:46:14.422896 6703 raft_consensus.cc:1261] T 93b7f7de1f8c49389c3d8c1e03c67f26 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:14.423069 4637 raft_consensus.cc:1261] T 93b7f7de1f8c49389c3d8c1e03c67f26 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEARNER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:14.423166 6967 raft_consensus.cc:1261] T 93b7f7de1f8c49389c3d8c1e03c67f26 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:14.423293 11071 consensus_queue.cc:1035] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:14.423614 10768 consensus_queue.cc:1035] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:14.424029 10768 consensus_queue.cc:1035] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:14.425359 11030 tablet.cc:1722] T 112cc778e5724fad8d4df9e9f88b4b9c P a4bf1df7fa6b4948be502c182bfbb833: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 20:46:14.425700 11030 tablet.cc:1620] T 112cc778e5724fad8d4df9e9f88b4b9c P a4bf1df7fa6b4948be502c182bfbb833: MemRowSet was empty: no flush needed.
I20240502 20:46:14.426659 11071 raft_consensus.cc:2932] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:14.427170 6703 raft_consensus.cc:2932] T 93b7f7de1f8c49389c3d8c1e03c67f26 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:14.428635 4637 raft_consensus.cc:2932] T 93b7f7de1f8c49389c3d8c1e03c67f26 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:14.431164 6967 raft_consensus.cc:2932] T 93b7f7de1f8c49389c3d8c1e03c67f26 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:14.431448 4231 catalog_manager.cc:5484] T 93b7f7de1f8c49389c3d8c1e03c67f26 P 7845c397230d49358ba2fad9db53f97c reported cstate change: config changed from index 5 to 6, 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c92e6e46782849479c8a0c6d82276fb2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:14.434494 4374 consensus_queue.cc:237] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } }
I20240502 20:46:14.435832 4637 raft_consensus.cc:1261] T 93b7f7de1f8c49389c3d8c1e03c67f26 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:14.435922 6703 raft_consensus.cc:1261] T 93b7f7de1f8c49389c3d8c1e03c67f26 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:14.436246 10768 consensus_queue.cc:1035] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:14.436444 11070 consensus_queue.cc:1035] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:14.438251 8185 raft_consensus.cc:2932] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:14.438568 4637 raft_consensus.cc:2932] T 93b7f7de1f8c49389c3d8c1e03c67f26 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:14.439877 4222 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 93b7f7de1f8c49389c3d8c1e03c67f26 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:14.440295 6703 raft_consensus.cc:2932] T 93b7f7de1f8c49389c3d8c1e03c67f26 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:14.441172 4230 catalog_manager.cc:5484] T 93b7f7de1f8c49389c3d8c1e03c67f26 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index 6 to 7, VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) evicted. New cstate: current_term: 1 leader_uuid: "c92e6e46782849479c8a0c6d82276fb2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: false } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:14.442869 10504 raft_consensus.cc:1050] T ac57ad308b174217ac74c4cb5e1c7285 P 7845c397230d49358ba2fad9db53f97c: attempting to promote NON_VOTER c92e6e46782849479c8a0c6d82276fb2 to VOTER
I20240502 20:46:14.443253 10504 consensus_queue.cc:237] T ac57ad308b174217ac74c4cb5e1c7285 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: false } }
I20240502 20:46:14.444622 4374 raft_consensus.cc:1261] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEARNER]: Refusing update from remote peer 7845c397230d49358ba2fad9db53f97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:14.445173 4507 raft_consensus.cc:1261] T ac57ad308b174217ac74c4cb5e1c7285 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer 7845c397230d49358ba2fad9db53f97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:14.445381 7099 raft_consensus.cc:1261] T ac57ad308b174217ac74c4cb5e1c7285 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Refusing update from remote peer 7845c397230d49358ba2fad9db53f97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:14.445804 9105 consensus_queue.cc:1035] T ac57ad308b174217ac74c4cb5e1c7285 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:46:14.446249 9105 consensus_queue.cc:1035] T ac57ad308b174217ac74c4cb5e1c7285 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:46:14.446473 10270 consensus_queue.cc:1035] T ac57ad308b174217ac74c4cb5e1c7285 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:46:14.447942 6945 tablet_service.cc:1514] Processing DeleteTablet for tablet 93b7f7de1f8c49389c3d8c1e03c67f26 with delete_type TABLET_DATA_TOMBSTONED (TS a47e5c56f13843018ac584914a8d4707 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:45156
I20240502 20:46:14.448174 10926 tablet_replica.cc:330] T 93b7f7de1f8c49389c3d8c1e03c67f26 P a47e5c56f13843018ac584914a8d4707: stopping tablet replica
I20240502 20:46:14.448311 10926 raft_consensus.cc:2229] T 93b7f7de1f8c49389c3d8c1e03c67f26 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:14.448483 10926 raft_consensus.cc:2258] T 93b7f7de1f8c49389c3d8c1e03c67f26 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:14.448637 9929 consensus_queue.cc:1035] T 71537846373844cc9ef396b79af60db4 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:14.449066 10926 ts_tablet_manager.cc:1875] T 93b7f7de1f8c49389c3d8c1e03c67f26 P a47e5c56f13843018ac584914a8d4707: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:14.449697 9105 raft_consensus.cc:2932] T ac57ad308b174217ac74c4cb5e1c7285 P 7845c397230d49358ba2fad9db53f97c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: false } } }
I20240502 20:46:14.451221 4507 raft_consensus.cc:2932] T ac57ad308b174217ac74c4cb5e1c7285 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: false } } }
I20240502 20:46:14.451432 10926 ts_tablet_manager.cc:1888] T 93b7f7de1f8c49389c3d8c1e03c67f26 P a47e5c56f13843018ac584914a8d4707: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 20:46:14.451503 10926 log.cc:1196] T 93b7f7de1f8c49389c3d8c1e03c67f26 P a47e5c56f13843018ac584914a8d4707: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/wal/wals/93b7f7de1f8c49389c3d8c1e03c67f26
I20240502 20:46:14.452111 4222 catalog_manager.cc:4830] TS a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): tablet 93b7f7de1f8c49389c3d8c1e03c67f26 (table rebalancer_extra_table_8 [id=553d578c620843f4b11d547972b8d776]) successfully deleted
I20240502 20:46:14.452204 7099 raft_consensus.cc:2932] T ac57ad308b174217ac74c4cb5e1c7285 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: false } } }
I20240502 20:46:14.454177 4374 raft_consensus.cc:2932] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: false } } }
I20240502 20:46:14.454267 4230 catalog_manager.cc:5484] T ac57ad308b174217ac74c4cb5e1c7285 P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: config changed from index 2 to 3, c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7845c397230d49358ba2fad9db53f97c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: false } } }
I20240502 20:46:14.459939 6703 consensus_queue.cc:237] T ac57ad308b174217ac74c4cb5e1c7285 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: false } }
I20240502 20:46:14.461055 7098 raft_consensus.cc:1261] T ac57ad308b174217ac74c4cb5e1c7285 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Refusing update from remote peer 7845c397230d49358ba2fad9db53f97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:14.461055 4374 raft_consensus.cc:1261] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer 7845c397230d49358ba2fad9db53f97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:14.461171 9929 consensus_queue.cc:1035] T 71537846373844cc9ef396b79af60db4 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:14.461623 10922 consensus_queue.cc:1035] T ac57ad308b174217ac74c4cb5e1c7285 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:14.462009 10922 consensus_queue.cc:1035] T ac57ad308b174217ac74c4cb5e1c7285 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:14.463789 11080 raft_consensus.cc:2932] T ac57ad308b174217ac74c4cb5e1c7285 P 7845c397230d49358ba2fad9db53f97c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: false } } }
I20240502 20:46:14.464298 7098 raft_consensus.cc:2932] T ac57ad308b174217ac74c4cb5e1c7285 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: false } } }
I20240502 20:46:14.464567 4374 raft_consensus.cc:2932] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: false } } }
I20240502 20:46:14.465487 4221 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ac57ad308b174217ac74c4cb5e1c7285 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:14.467542 4227 catalog_manager.cc:5484] T ac57ad308b174217ac74c4cb5e1c7285 P 201d2b6485084e7dad2cbb722af94779 reported cstate change: config changed from index 3 to 4, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New cstate: current_term: 1 leader_uuid: "7845c397230d49358ba2fad9db53f97c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: false } } }
I20240502 20:46:14.470108 4487 tablet_service.cc:1514] Processing DeleteTablet for tablet ac57ad308b174217ac74c4cb5e1c7285 with delete_type TABLET_DATA_TOMBSTONED (TS c3d26eafd51547acb780d4aac5ebe7cc not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48890
I20240502 20:46:14.470502 11083 tablet_replica.cc:330] T ac57ad308b174217ac74c4cb5e1c7285 P c3d26eafd51547acb780d4aac5ebe7cc: stopping tablet replica
I20240502 20:46:14.470654 11083 raft_consensus.cc:2229] T ac57ad308b174217ac74c4cb5e1c7285 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:14.470968 11083 raft_consensus.cc:2258] T ac57ad308b174217ac74c4cb5e1c7285 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:14.471848 11083 ts_tablet_manager.cc:1875] T ac57ad308b174217ac74c4cb5e1c7285 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:14.474488 11083 ts_tablet_manager.cc:1888] T ac57ad308b174217ac74c4cb5e1c7285 P c3d26eafd51547acb780d4aac5ebe7cc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 20:46:14.474558 11083 log.cc:1196] T ac57ad308b174217ac74c4cb5e1c7285 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal/wals/ac57ad308b174217ac74c4cb5e1c7285
I20240502 20:46:14.475159 4221 catalog_manager.cc:4830] TS c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097): tablet ac57ad308b174217ac74c4cb5e1c7285 (table rebalancer_extra_table_10 [id=c8bf855c4f414063abc8d3c0e2880292]) successfully deleted
I20240502 20:46:14.527288 4231 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 20:46:14.533121 10757 tablet.cc:1722] T 112cc778e5724fad8d4df9e9f88b4b9c P 201d2b6485084e7dad2cbb722af94779: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 20:46:14.533639 11030 tablet.cc:1722] T 112cc778e5724fad8d4df9e9f88b4b9c P a4bf1df7fa6b4948be502c182bfbb833: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 20:46:14.533639 11025 tablet.cc:1722] T 112cc778e5724fad8d4df9e9f88b4b9c P a47e5c56f13843018ac584914a8d4707: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 20:46:14.535638 4231 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41812:
name: "rebalancer_extra_table_20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:46:14.540418 4227 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: rebalancer_dynamic_table_new_name [id=39fbb150e9d84b52a68efaf1efbc90a9] alter complete (version 3)
I20240502 20:46:14.542387 6815 tablet_service.cc:1467] Processing CreateTablet for tablet 6201083694c344c8ad63720be89a3cdb (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=07b224e518b3432b87d206a643db028f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:14.542582 6802 tablet_service.cc:1467] Processing CreateTablet for tablet 77b294bcc497479e8e8bd8e7f9cc1ff2 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=07b224e518b3432b87d206a643db028f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:14.542712 6815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6201083694c344c8ad63720be89a3cdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:14.542866 6802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77b294bcc497479e8e8bd8e7f9cc1ff2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:14.543134 6945 tablet_service.cc:1467] Processing CreateTablet for tablet 6201083694c344c8ad63720be89a3cdb (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=07b224e518b3432b87d206a643db028f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:14.543402 6945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6201083694c344c8ad63720be89a3cdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:14.544934 10742 tablet_bootstrap.cc:492] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap starting.
I20240502 20:46:14.545696 11089 tablet_bootstrap.cc:492] T 6201083694c344c8ad63720be89a3cdb P a47e5c56f13843018ac584914a8d4707: Bootstrap starting.
I20240502 20:46:14.546005 10742 tablet_bootstrap.cc:654] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P a4bf1df7fa6b4948be502c182bfbb833: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:14.545964 4750 tablet_service.cc:1467] Processing CreateTablet for tablet 77b294bcc497479e8e8bd8e7f9cc1ff2 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=07b224e518b3432b87d206a643db028f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:14.546252 4750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77b294bcc497479e8e8bd8e7f9cc1ff2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:14.546255 6815 tablet_service.cc:1467] Processing CreateTablet for tablet 6ff3c95ccfbf486fbc4bfc636afa829b (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=07b224e518b3432b87d206a643db028f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:14.546460 6815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ff3c95ccfbf486fbc4bfc636afa829b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:14.549048 11092 tablet_bootstrap.cc:492] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:46:14.548996 6681 tablet_service.cc:1467] Processing CreateTablet for tablet 6ff3c95ccfbf486fbc4bfc636afa829b (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=07b224e518b3432b87d206a643db028f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:14.549106 11089 tablet_bootstrap.cc:654] T 6201083694c344c8ad63720be89a3cdb P a47e5c56f13843018ac584914a8d4707: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:14.549296 6681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ff3c95ccfbf486fbc4bfc636afa829b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:14.550313 11089 tablet_bootstrap.cc:492] T 6201083694c344c8ad63720be89a3cdb P a47e5c56f13843018ac584914a8d4707: No bootstrap required, opened a new log
I20240502 20:46:14.550406 11089 ts_tablet_manager.cc:1367] T 6201083694c344c8ad63720be89a3cdb P a47e5c56f13843018ac584914a8d4707: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240502 20:46:14.550839 11089 raft_consensus.cc:348] T 6201083694c344c8ad63720be89a3cdb P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:14.550969 11089 raft_consensus.cc:374] T 6201083694c344c8ad63720be89a3cdb P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:14.551012 11089 raft_consensus.cc:729] T 6201083694c344c8ad63720be89a3cdb P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Initialized, Role: FOLLOWER
I20240502 20:46:14.551156 11089 consensus_queue.cc:260] T 6201083694c344c8ad63720be89a3cdb P a47e5c56f13843018ac584914a8d4707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:14.551362 11092 tablet_bootstrap.cc:654] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:14.551487 11089 ts_tablet_manager.cc:1398] T 6201083694c344c8ad63720be89a3cdb P a47e5c56f13843018ac584914a8d4707: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:14.552743 7079 tablet_service.cc:1467] Processing CreateTablet for tablet 6201083694c344c8ad63720be89a3cdb (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=07b224e518b3432b87d206a643db028f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:14.552881 7078 tablet_service.cc:1467] Processing CreateTablet for tablet 6ff3c95ccfbf486fbc4bfc636afa829b (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=07b224e518b3432b87d206a643db028f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:14.553052 7079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6201083694c344c8ad63720be89a3cdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:14.553095 7077 tablet_service.cc:1467] Processing CreateTablet for tablet 77b294bcc497479e8e8bd8e7f9cc1ff2 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=07b224e518b3432b87d206a643db028f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:14.553207 7078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ff3c95ccfbf486fbc4bfc636afa829b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:14.553373 7077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77b294bcc497479e8e8bd8e7f9cc1ff2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:14.554121 11092 tablet_bootstrap.cc:492] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:46:14.554206 11092 ts_tablet_manager.cc:1367] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 20:46:14.554625 11092 raft_consensus.cc:348] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:14.554720 11092 raft_consensus.cc:374] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:14.554761 11092 raft_consensus.cc:729] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:46:14.554881 11092 consensus_queue.cc:260] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:14.555176 11092 ts_tablet_manager.cc:1398] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:14.556864 10742 tablet_bootstrap.cc:492] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P a4bf1df7fa6b4948be502c182bfbb833: No bootstrap required, opened a new log
I20240502 20:46:14.556957 10742 ts_tablet_manager.cc:1367] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P a4bf1df7fa6b4948be502c182bfbb833: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240502 20:46:14.557471 10742 raft_consensus.cc:348] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:14.557596 10742 raft_consensus.cc:374] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:14.557638 10742 raft_consensus.cc:729] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Initialized, Role: FOLLOWER
I20240502 20:46:14.557781 10742 consensus_queue.cc:260] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P a4bf1df7fa6b4948be502c182bfbb833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:14.558063 10742 ts_tablet_manager.cc:1398] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P a4bf1df7fa6b4948be502c182bfbb833: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:14.559114 10742 tablet_bootstrap.cc:492] T 6201083694c344c8ad63720be89a3cdb P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap starting.
I20240502 20:46:14.559648 11094 tablet_bootstrap.cc:492] T 6ff3c95ccfbf486fbc4bfc636afa829b P 7845c397230d49358ba2fad9db53f97c: Bootstrap starting.
I20240502 20:46:14.561041 11094 tablet_bootstrap.cc:654] T 6ff3c95ccfbf486fbc4bfc636afa829b P 7845c397230d49358ba2fad9db53f97c: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:14.561117 10742 tablet_bootstrap.cc:654] T 6201083694c344c8ad63720be89a3cdb P a4bf1df7fa6b4948be502c182bfbb833: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:14.561805 11096 tablet_bootstrap.cc:492] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P 201d2b6485084e7dad2cbb722af94779: Bootstrap starting.
I20240502 20:46:14.562711 10742 tablet_bootstrap.cc:492] T 6201083694c344c8ad63720be89a3cdb P a4bf1df7fa6b4948be502c182bfbb833: No bootstrap required, opened a new log
I20240502 20:46:14.562804 10742 ts_tablet_manager.cc:1367] T 6201083694c344c8ad63720be89a3cdb P a4bf1df7fa6b4948be502c182bfbb833: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 20:46:14.563397 10742 raft_consensus.cc:348] T 6201083694c344c8ad63720be89a3cdb P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:14.563500 10742 raft_consensus.cc:374] T 6201083694c344c8ad63720be89a3cdb P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:14.563534 10742 raft_consensus.cc:729] T 6201083694c344c8ad63720be89a3cdb P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Initialized, Role: FOLLOWER
I20240502 20:46:14.563632 10742 consensus_queue.cc:260] T 6201083694c344c8ad63720be89a3cdb P a4bf1df7fa6b4948be502c182bfbb833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:14.563903 10742 ts_tablet_manager.cc:1398] T 6201083694c344c8ad63720be89a3cdb P a4bf1df7fa6b4948be502c182bfbb833: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:14.562878 11096 tablet_bootstrap.cc:654] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P 201d2b6485084e7dad2cbb722af94779: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:14.564049 10742 tablet_bootstrap.cc:492] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap starting.
I20240502 20:46:14.564247 11094 tablet_bootstrap.cc:492] T 6ff3c95ccfbf486fbc4bfc636afa829b P 7845c397230d49358ba2fad9db53f97c: No bootstrap required, opened a new log
I20240502 20:46:14.564719 11094 ts_tablet_manager.cc:1367] T 6ff3c95ccfbf486fbc4bfc636afa829b P 7845c397230d49358ba2fad9db53f97c: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240502 20:46:14.565237 11094 raft_consensus.cc:348] T 6ff3c95ccfbf486fbc4bfc636afa829b P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:14.565317 10742 tablet_bootstrap.cc:654] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:14.565330 11096 tablet_bootstrap.cc:492] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P 201d2b6485084e7dad2cbb722af94779: No bootstrap required, opened a new log
I20240502 20:46:14.565338 11094 raft_consensus.cc:374] T 6ff3c95ccfbf486fbc4bfc636afa829b P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:14.565369 11094 raft_consensus.cc:729] T 6ff3c95ccfbf486fbc4bfc636afa829b P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Initialized, Role: FOLLOWER
I20240502 20:46:14.565410 11096 ts_tablet_manager.cc:1367] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P 201d2b6485084e7dad2cbb722af94779: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 20:46:14.565510 11094 consensus_queue.cc:260] T 6ff3c95ccfbf486fbc4bfc636afa829b P 7845c397230d49358ba2fad9db53f97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:14.565901 11094 ts_tablet_manager.cc:1398] T 6ff3c95ccfbf486fbc4bfc636afa829b P 7845c397230d49358ba2fad9db53f97c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:14.566118 11096 raft_consensus.cc:348] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:14.566200 11096 raft_consensus.cc:374] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:14.566228 11096 raft_consensus.cc:729] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Initialized, Role: FOLLOWER
I20240502 20:46:14.566347 11096 consensus_queue.cc:260] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P 201d2b6485084e7dad2cbb722af94779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:14.566478 4507 raft_consensus.cc:1261] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P c3d26eafd51547acb780d4aac5ebe7cc [term 2 FOLLOWER]: Refusing update from remote peer a4bf1df7fa6b4948be502c182bfbb833: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 20:46:14.566638 11096 ts_tablet_manager.cc:1398] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P 201d2b6485084e7dad2cbb722af94779: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:14.566767 11096 tablet_bootstrap.cc:492] T 6ff3c95ccfbf486fbc4bfc636afa829b P 201d2b6485084e7dad2cbb722af94779: Bootstrap starting.
I20240502 20:46:14.566965 10742 tablet_bootstrap.cc:492] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833: No bootstrap required, opened a new log
I20240502 20:46:14.567056 10742 ts_tablet_manager.cc:1367] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:46:14.567584 10742 raft_consensus.cc:348] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:14.567716 10742 raft_consensus.cc:374] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:14.567761 10742 raft_consensus.cc:729] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Initialized, Role: FOLLOWER
I20240502 20:46:14.567859 10742 consensus_queue.cc:260] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:14.568055 11096 tablet_bootstrap.cc:654] T 6ff3c95ccfbf486fbc4bfc636afa829b P 201d2b6485084e7dad2cbb722af94779: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:14.568112 10742 ts_tablet_manager.cc:1398] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:14.567584 10776 consensus_queue.cc:1035] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20240502 20:46:14.569191 11096 tablet_bootstrap.cc:492] T 6ff3c95ccfbf486fbc4bfc636afa829b P 201d2b6485084e7dad2cbb722af94779: No bootstrap required, opened a new log
I20240502 20:46:14.569267 11096 ts_tablet_manager.cc:1367] T 6ff3c95ccfbf486fbc4bfc636afa829b P 201d2b6485084e7dad2cbb722af94779: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:46:14.569669 11096 raft_consensus.cc:348] T 6ff3c95ccfbf486fbc4bfc636afa829b P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:14.569861 11096 raft_consensus.cc:374] T 6ff3c95ccfbf486fbc4bfc636afa829b P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:14.569953 11096 raft_consensus.cc:729] T 6ff3c95ccfbf486fbc4bfc636afa829b P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Initialized, Role: FOLLOWER
I20240502 20:46:14.570140 11096 consensus_queue.cc:260] T 6ff3c95ccfbf486fbc4bfc636afa829b P 201d2b6485084e7dad2cbb722af94779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:14.571012 11096 ts_tablet_manager.cc:1398] T 6ff3c95ccfbf486fbc4bfc636afa829b P 201d2b6485084e7dad2cbb722af94779: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:14.571220 11096 tablet_bootstrap.cc:492] T 6201083694c344c8ad63720be89a3cdb P 201d2b6485084e7dad2cbb722af94779: Bootstrap starting.
I20240502 20:46:14.572157 11096 tablet_bootstrap.cc:654] T 6201083694c344c8ad63720be89a3cdb P 201d2b6485084e7dad2cbb722af94779: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:14.579576 11096 tablet_bootstrap.cc:492] T 6201083694c344c8ad63720be89a3cdb P 201d2b6485084e7dad2cbb722af94779: No bootstrap required, opened a new log
I20240502 20:46:14.579674 11096 ts_tablet_manager.cc:1367] T 6201083694c344c8ad63720be89a3cdb P 201d2b6485084e7dad2cbb722af94779: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240502 20:46:14.580122 11096 raft_consensus.cc:348] T 6201083694c344c8ad63720be89a3cdb P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:14.580240 11096 raft_consensus.cc:374] T 6201083694c344c8ad63720be89a3cdb P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:14.580283 11096 raft_consensus.cc:729] T 6201083694c344c8ad63720be89a3cdb P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Initialized, Role: FOLLOWER
I20240502 20:46:14.580406 11096 consensus_queue.cc:260] T 6201083694c344c8ad63720be89a3cdb P 201d2b6485084e7dad2cbb722af94779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:14.580662 11096 ts_tablet_manager.cc:1398] T 6201083694c344c8ad63720be89a3cdb P 201d2b6485084e7dad2cbb722af94779: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:14.591961 10922 raft_consensus.cc:482] T 6ff3c95ccfbf486fbc4bfc636afa829b P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:14.592061 10922 raft_consensus.cc:504] T 6ff3c95ccfbf486fbc4bfc636afa829b P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:14.592453 10922 leader_election.cc:288] T 6ff3c95ccfbf486fbc4bfc636afa829b P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643), a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867)
I20240502 20:46:14.592779 7098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ff3c95ccfbf486fbc4bfc636afa829b" candidate_uuid: "7845c397230d49358ba2fad9db53f97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201d2b6485084e7dad2cbb722af94779" is_pre_election: true
I20240502 20:46:14.592895 7098 raft_consensus.cc:2454] T 6ff3c95ccfbf486fbc4bfc636afa829b P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7845c397230d49358ba2fad9db53f97c in term 0.
I20240502 20:46:14.592952 6835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ff3c95ccfbf486fbc4bfc636afa829b" candidate_uuid: "7845c397230d49358ba2fad9db53f97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833" is_pre_election: true
I20240502 20:46:14.593104 6835 raft_consensus.cc:2454] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7845c397230d49358ba2fad9db53f97c in term 0.
I20240502 20:46:14.593381 6636 leader_election.cc:302] T 6ff3c95ccfbf486fbc4bfc636afa829b P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201d2b6485084e7dad2cbb722af94779, 7845c397230d49358ba2fad9db53f97c; no voters:
I20240502 20:46:14.593601 10922 raft_consensus.cc:2781] T 6ff3c95ccfbf486fbc4bfc636afa829b P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:14.593664 10922 raft_consensus.cc:482] T 6ff3c95ccfbf486fbc4bfc636afa829b P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:14.593731 10922 raft_consensus.cc:3037] T 6ff3c95ccfbf486fbc4bfc636afa829b P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:14.594599 10922 raft_consensus.cc:504] T 6ff3c95ccfbf486fbc4bfc636afa829b P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:14.594894 10922 leader_election.cc:288] T 6ff3c95ccfbf486fbc4bfc636afa829b P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 election: Requested vote from peers 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643), a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867)
I20240502 20:46:14.595180 7098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ff3c95ccfbf486fbc4bfc636afa829b" candidate_uuid: "7845c397230d49358ba2fad9db53f97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201d2b6485084e7dad2cbb722af94779"
I20240502 20:46:14.595289 7098 raft_consensus.cc:3037] T 6ff3c95ccfbf486fbc4bfc636afa829b P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:14.595237 6835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ff3c95ccfbf486fbc4bfc636afa829b" candidate_uuid: "7845c397230d49358ba2fad9db53f97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833"
I20240502 20:46:14.595378 6835 raft_consensus.cc:3037] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:14.596264 7098 raft_consensus.cc:2454] T 6ff3c95ccfbf486fbc4bfc636afa829b P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7845c397230d49358ba2fad9db53f97c in term 1.
I20240502 20:46:14.596355 6835 raft_consensus.cc:2454] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7845c397230d49358ba2fad9db53f97c in term 1.
I20240502 20:46:14.596635 6636 leader_election.cc:302] T 6ff3c95ccfbf486fbc4bfc636afa829b P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201d2b6485084e7dad2cbb722af94779, 7845c397230d49358ba2fad9db53f97c; no voters:
I20240502 20:46:14.597024 10922 raft_consensus.cc:2781] T 6ff3c95ccfbf486fbc4bfc636afa829b P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:14.597123 10922 raft_consensus.cc:686] T 6ff3c95ccfbf486fbc4bfc636afa829b P 7845c397230d49358ba2fad9db53f97c [term 1 LEADER]: Becoming Leader. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Running, Role: LEADER
I20240502 20:46:14.597270 10922 consensus_queue.cc:237] T 6ff3c95ccfbf486fbc4bfc636afa829b P 7845c397230d49358ba2fad9db53f97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:14.599591 4228 catalog_manager.cc:5484] T 6ff3c95ccfbf486fbc4bfc636afa829b P 7845c397230d49358ba2fad9db53f97c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7845c397230d49358ba2fad9db53f97c (127.30.32.69). New cstate: current_term: 1 leader_uuid: "7845c397230d49358ba2fad9db53f97c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } health_report { overall_health: HEALTHY } } }
I20240502 20:46:14.603432 10917 raft_consensus.cc:1050] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc: attempting to promote NON_VOTER 201d2b6485084e7dad2cbb722af94779 to VOTER
I20240502 20:46:14.603870 10917 consensus_queue.cc:237] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } }
I20240502 20:46:14.605448 7098 raft_consensus.cc:1261] T d37730b109d04b6395f75c7552524877 P 201d2b6485084e7dad2cbb722af94779 [term 1 LEARNER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:14.605700 4637 raft_consensus.cc:1261] T d37730b109d04b6395f75c7552524877 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:14.606487 11135 consensus_queue.cc:1035] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:14.606738 6835 raft_consensus.cc:1261] T d37730b109d04b6395f75c7552524877 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:14.606940 10873 consensus_queue.cc:1035] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:14.607373 10873 consensus_queue.cc:1035] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:14.609910 11135 raft_consensus.cc:2932] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:14.610745 7098 raft_consensus.cc:2932] T d37730b109d04b6395f75c7552524877 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:14.612058 4637 raft_consensus.cc:2932] T d37730b109d04b6395f75c7552524877 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:14.614351 4235 catalog_manager.cc:5484] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: config changed from index 7 to 8, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:14.614666 6835 raft_consensus.cc:2932] T d37730b109d04b6395f75c7552524877 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:14.625739 6702 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a9afb75bfc114f59850eaea084a306f0"
dest_uuid: "7845c397230d49358ba2fad9db53f97c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52210
I20240502 20:46:14.625866 6702 raft_consensus.cc:595] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c [term 1 LEADER]: Received request to transfer leadership
I20240502 20:46:14.642738 4230 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 20:46:14.642915 4230 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 20:46:14.645179 4230 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: Sending DeleteTablet for 3 replicas of tablet 112cc778e5724fad8d4df9e9f88b4b9c
I20240502 20:46:14.645861 6945 tablet_service.cc:1514] Processing DeleteTablet for tablet 112cc778e5724fad8d4df9e9f88b4b9c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:14 UTC) from {username='slave'} at 127.0.0.1:45156
I20240502 20:46:14.645861 7079 tablet_service.cc:1514] Processing DeleteTablet for tablet 112cc778e5724fad8d4df9e9f88b4b9c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:14 UTC) from {username='slave'} at 127.0.0.1:59162
I20240502 20:46:14.646068 10926 tablet_replica.cc:330] T 112cc778e5724fad8d4df9e9f88b4b9c P a47e5c56f13843018ac584914a8d4707: stopping tablet replica
I20240502 20:46:14.646201 11164 tablet_replica.cc:330] T 112cc778e5724fad8d4df9e9f88b4b9c P 201d2b6485084e7dad2cbb722af94779: stopping tablet replica
I20240502 20:46:14.646217 10926 raft_consensus.cc:2229] T 112cc778e5724fad8d4df9e9f88b4b9c P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:14.646327 11164 raft_consensus.cc:2229] T 112cc778e5724fad8d4df9e9f88b4b9c P 201d2b6485084e7dad2cbb722af94779 [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:14.646373 10926 raft_consensus.cc:2258] T 112cc778e5724fad8d4df9e9f88b4b9c P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:14.646538 11164 raft_consensus.cc:2258] T 112cc778e5724fad8d4df9e9f88b4b9c P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:14.646979 6815 tablet_service.cc:1514] Processing DeleteTablet for tablet 112cc778e5724fad8d4df9e9f88b4b9c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:14 UTC) from {username='slave'} at 127.0.0.1:35290
I20240502 20:46:14.647176 11164 ts_tablet_manager.cc:1875] T 112cc778e5724fad8d4df9e9f88b4b9c P 201d2b6485084e7dad2cbb722af94779: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:14.647221 10926 ts_tablet_manager.cc:1875] T 112cc778e5724fad8d4df9e9f88b4b9c P a47e5c56f13843018ac584914a8d4707: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:14.647344 11165 tablet_replica.cc:330] T 112cc778e5724fad8d4df9e9f88b4b9c P a4bf1df7fa6b4948be502c182bfbb833: stopping tablet replica
I20240502 20:46:14.647485 11165 raft_consensus.cc:2229] T 112cc778e5724fad8d4df9e9f88b4b9c P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:14.647603 11165 raft_consensus.cc:2258] T 112cc778e5724fad8d4df9e9f88b4b9c P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:14.648228 11165 ts_tablet_manager.cc:1875] T 112cc778e5724fad8d4df9e9f88b4b9c P a4bf1df7fa6b4948be502c182bfbb833: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:14.649152 10926 ts_tablet_manager.cc:1888] T 112cc778e5724fad8d4df9e9f88b4b9c P a47e5c56f13843018ac584914a8d4707: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:14.649204 10926 log.cc:1196] T 112cc778e5724fad8d4df9e9f88b4b9c P a47e5c56f13843018ac584914a8d4707: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/wal/wals/112cc778e5724fad8d4df9e9f88b4b9c
I20240502 20:46:14.649406 10926 ts_tablet_manager.cc:1909] T 112cc778e5724fad8d4df9e9f88b4b9c P a47e5c56f13843018ac584914a8d4707: Deleting consensus metadata
I20240502 20:46:14.649495 11164 ts_tablet_manager.cc:1888] T 112cc778e5724fad8d4df9e9f88b4b9c P 201d2b6485084e7dad2cbb722af94779: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:14.649586 11164 log.cc:1196] T 112cc778e5724fad8d4df9e9f88b4b9c P 201d2b6485084e7dad2cbb722af94779: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/wal/wals/112cc778e5724fad8d4df9e9f88b4b9c
I20240502 20:46:14.649864 11164 ts_tablet_manager.cc:1909] T 112cc778e5724fad8d4df9e9f88b4b9c P 201d2b6485084e7dad2cbb722af94779: Deleting consensus metadata
I20240502 20:46:14.650271 4222 catalog_manager.cc:4830] TS a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): tablet 112cc778e5724fad8d4df9e9f88b4b9c (table rebalancer_dynamic_table_new_name [id=39fbb150e9d84b52a68efaf1efbc90a9]) successfully deleted
I20240502 20:46:14.650491 4219 catalog_manager.cc:4830] TS 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): tablet 112cc778e5724fad8d4df9e9f88b4b9c (table rebalancer_dynamic_table_new_name [id=39fbb150e9d84b52a68efaf1efbc90a9]) successfully deleted
I20240502 20:46:14.651042 11165 ts_tablet_manager.cc:1888] T 112cc778e5724fad8d4df9e9f88b4b9c P a4bf1df7fa6b4948be502c182bfbb833: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:14.651110 11165 log.cc:1196] T 112cc778e5724fad8d4df9e9f88b4b9c P a4bf1df7fa6b4948be502c182bfbb833: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/wal/wals/112cc778e5724fad8d4df9e9f88b4b9c
I20240502 20:46:14.651392 11165 ts_tablet_manager.cc:1909] T 112cc778e5724fad8d4df9e9f88b4b9c P a4bf1df7fa6b4948be502c182bfbb833: Deleting consensus metadata
I20240502 20:46:14.652027 4220 catalog_manager.cc:4830] TS a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867): tablet 112cc778e5724fad8d4df9e9f88b4b9c (table rebalancer_dynamic_table_new_name [id=39fbb150e9d84b52a68efaf1efbc90a9]) successfully deleted
I20240502 20:46:14.655520 4507 consensus_queue.cc:237] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } }
I20240502 20:46:14.657299 6835 raft_consensus.cc:1261] T 45ade8c672d8445eaffb179772e3759a P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:14.657299 4637 raft_consensus.cc:1261] T 45ade8c672d8445eaffb179772e3759a P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:14.657953 10917 consensus_queue.cc:1035] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 20:46:14.658262 4444 consensus_peers.cc:485] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 45ade8c672d8445eaffb179772e3759a. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:14.658612 10917 consensus_queue.cc:1035] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:14.660058 10873 raft_consensus.cc:2932] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } } }
I20240502 20:46:14.660485 6831 raft_consensus.cc:2932] T 45ade8c672d8445eaffb179772e3759a P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } } }
I20240502 20:46:14.663151 4230 catalog_manager.cc:5484] T 45ade8c672d8445eaffb179772e3759a P a4bf1df7fa6b4948be502c182bfbb833 reported cstate change: config changed from index 6 to 7, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } } }
I20240502 20:46:14.663895 4637 raft_consensus.cc:2932] T 45ade8c672d8445eaffb179772e3759a P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } } }
I20240502 20:46:14.675535 9929 raft_consensus.cc:482] T 6201083694c344c8ad63720be89a3cdb P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:14.675644 9929 raft_consensus.cc:504] T 6201083694c344c8ad63720be89a3cdb P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:14.676041 9929 leader_election.cc:288] T 6201083694c344c8ad63720be89a3cdb P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
I20240502 20:46:14.676466 7098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6201083694c344c8ad63720be89a3cdb" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201d2b6485084e7dad2cbb722af94779" is_pre_election: true
I20240502 20:46:14.676519 6967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6201083694c344c8ad63720be89a3cdb" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47e5c56f13843018ac584914a8d4707" is_pre_election: true
I20240502 20:46:14.676605 7098 raft_consensus.cc:2454] T 6201083694c344c8ad63720be89a3cdb P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 0.
I20240502 20:46:14.676636 6967 raft_consensus.cc:2454] T 6201083694c344c8ad63720be89a3cdb P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 0.
I20240502 20:46:14.676909 6771 leader_election.cc:302] T 6201083694c344c8ad63720be89a3cdb P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a47e5c56f13843018ac584914a8d4707, a4bf1df7fa6b4948be502c182bfbb833; no voters:
I20240502 20:46:14.677142 9929 raft_consensus.cc:2781] T 6201083694c344c8ad63720be89a3cdb P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:14.677197 9929 raft_consensus.cc:482] T 6201083694c344c8ad63720be89a3cdb P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:14.677248 9929 raft_consensus.cc:3037] T 6201083694c344c8ad63720be89a3cdb P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:14.678135 9929 raft_consensus.cc:504] T 6201083694c344c8ad63720be89a3cdb P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:14.678411 9929 leader_election.cc:288] T 6201083694c344c8ad63720be89a3cdb P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 election: Requested vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
I20240502 20:46:14.678825 7098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6201083694c344c8ad63720be89a3cdb" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201d2b6485084e7dad2cbb722af94779"
I20240502 20:46:14.678895 6967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6201083694c344c8ad63720be89a3cdb" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47e5c56f13843018ac584914a8d4707"
I20240502 20:46:14.678961 7098 raft_consensus.cc:3037] T 6201083694c344c8ad63720be89a3cdb P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:14.679014 6967 raft_consensus.cc:3037] T 6201083694c344c8ad63720be89a3cdb P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:14.679934 7098 raft_consensus.cc:2454] T 6201083694c344c8ad63720be89a3cdb P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 1.
I20240502 20:46:14.679950 6967 raft_consensus.cc:2454] T 6201083694c344c8ad63720be89a3cdb P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 1.
I20240502 20:46:14.680251 6768 leader_election.cc:302] T 6201083694c344c8ad63720be89a3cdb P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201d2b6485084e7dad2cbb722af94779, a4bf1df7fa6b4948be502c182bfbb833; no voters:
I20240502 20:46:14.680441 9929 raft_consensus.cc:2781] T 6201083694c344c8ad63720be89a3cdb P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:14.680548 9929 raft_consensus.cc:686] T 6201083694c344c8ad63720be89a3cdb P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEADER]: Becoming Leader. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Running, Role: LEADER
I20240502 20:46:14.680694 9929 consensus_queue.cc:237] T 6201083694c344c8ad63720be89a3cdb P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:14.683239 4228 catalog_manager.cc:5484] T 6201083694c344c8ad63720be89a3cdb P a4bf1df7fa6b4948be502c182bfbb833 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70). New cstate: current_term: 1 leader_uuid: "a4bf1df7fa6b4948be502c182bfbb833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:14.712222 10922 raft_consensus.cc:979] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c: : Instructing follower a4bf1df7fa6b4948be502c182bfbb833 to start an election
I20240502 20:46:14.712352 10922 raft_consensus.cc:1067] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c [term 1 LEADER]: Signalling peer a4bf1df7fa6b4948be502c182bfbb833 to start an election
I20240502 20:46:14.713030 6831 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a9afb75bfc114f59850eaea084a306f0"
dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833"
from {username='slave'} at 127.30.32.69:38627
I20240502 20:46:14.713167 6831 raft_consensus.cc:482] T a9afb75bfc114f59850eaea084a306f0 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:46:14.713229 6831 raft_consensus.cc:3037] T a9afb75bfc114f59850eaea084a306f0 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:14.714376 6831 raft_consensus.cc:504] T a9afb75bfc114f59850eaea084a306f0 P a4bf1df7fa6b4948be502c182bfbb833 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } }
I20240502 20:46:14.714815 6831 leader_election.cc:288] T a9afb75bfc114f59850eaea084a306f0 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 election: Requested vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:46:14.715070 6702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9afb75bfc114f59850eaea084a306f0" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "7845c397230d49358ba2fad9db53f97c"
I20240502 20:46:14.715207 6702 raft_consensus.cc:3032] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:46:14.715250 6702 raft_consensus.cc:729] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Running, Role: LEADER
I20240502 20:46:14.715400 6967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9afb75bfc114f59850eaea084a306f0" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "a47e5c56f13843018ac584914a8d4707"
I20240502 20:46:14.715417 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9afb75bfc114f59850eaea084a306f0" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ed47c023452a497192f35d2f617bf9bd"
I20240502 20:46:14.715416 6702 consensus_queue.cc:260] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } }
I20240502 20:46:14.715502 6967 raft_consensus.cc:3037] T a9afb75bfc114f59850eaea084a306f0 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:14.715526 4768 raft_consensus.cc:3037] T a9afb75bfc114f59850eaea084a306f0 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:14.715626 6702 raft_consensus.cc:3037] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:14.716280 6967 raft_consensus.cc:2454] T a9afb75bfc114f59850eaea084a306f0 P a47e5c56f13843018ac584914a8d4707 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 2.
I20240502 20:46:14.716564 6702 raft_consensus.cc:2454] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 2.
I20240502 20:46:14.716578 4768 raft_consensus.cc:2454] T a9afb75bfc114f59850eaea084a306f0 P ed47c023452a497192f35d2f617bf9bd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 2.
I20240502 20:46:14.716871 6771 leader_election.cc:302] T a9afb75bfc114f59850eaea084a306f0 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: a47e5c56f13843018ac584914a8d4707, a4bf1df7fa6b4948be502c182bfbb833, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:46:14.717067 10776 raft_consensus.cc:2781] T a9afb75bfc114f59850eaea084a306f0 P a4bf1df7fa6b4948be502c182bfbb833 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:46:14.717165 10776 raft_consensus.cc:686] T a9afb75bfc114f59850eaea084a306f0 P a4bf1df7fa6b4948be502c182bfbb833 [term 2 LEADER]: Becoming Leader. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Running, Role: LEADER
I20240502 20:46:14.717324 10776 consensus_queue.cc:237] T a9afb75bfc114f59850eaea084a306f0 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } }
I20240502 20:46:14.719646 4228 catalog_manager.cc:5484] T a9afb75bfc114f59850eaea084a306f0 P a4bf1df7fa6b4948be502c182bfbb833 reported cstate change: term changed from 1 to 2, leader changed from 7845c397230d49358ba2fad9db53f97c (127.30.32.69) to a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70). New cstate: current_term: 2 leader_uuid: "a4bf1df7fa6b4948be502c182bfbb833" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:14.719828 4507 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d37730b109d04b6395f75c7552524877"
dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42880
I20240502 20:46:14.719950 4507 raft_consensus.cc:595] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Received request to transfer leadership
I20240502 20:46:14.747658 4228 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41812:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 20:46:14.752956 6945 tablet_service.cc:1467] Processing CreateTablet for tablet d8d1661cfeee47f8bda913c7a2ca90b0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=12a6e673f40f4d7e8dc0931e9521302c]), partition=
I20240502 20:46:14.753233 6815 tablet_service.cc:1467] Processing CreateTablet for tablet d8d1661cfeee47f8bda913c7a2ca90b0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=12a6e673f40f4d7e8dc0931e9521302c]), partition=
I20240502 20:46:14.753283 6945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8d1661cfeee47f8bda913c7a2ca90b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:14.753460 6815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8d1661cfeee47f8bda913c7a2ca90b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:14.755544 11089 tablet_bootstrap.cc:492] T d8d1661cfeee47f8bda913c7a2ca90b0 P a47e5c56f13843018ac584914a8d4707: Bootstrap starting.
I20240502 20:46:14.755544 10742 tablet_bootstrap.cc:492] T d8d1661cfeee47f8bda913c7a2ca90b0 P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap starting.
I20240502 20:46:14.756466 11089 tablet_bootstrap.cc:654] T d8d1661cfeee47f8bda913c7a2ca90b0 P a47e5c56f13843018ac584914a8d4707: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:14.756466 10742 tablet_bootstrap.cc:654] T d8d1661cfeee47f8bda913c7a2ca90b0 P a4bf1df7fa6b4948be502c182bfbb833: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:14.757305 6681 tablet_service.cc:1467] Processing CreateTablet for tablet d8d1661cfeee47f8bda913c7a2ca90b0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=12a6e673f40f4d7e8dc0931e9521302c]), partition=
I20240502 20:46:14.757551 6681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8d1661cfeee47f8bda913c7a2ca90b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:14.758844 11089 tablet_bootstrap.cc:492] T d8d1661cfeee47f8bda913c7a2ca90b0 P a47e5c56f13843018ac584914a8d4707: No bootstrap required, opened a new log
I20240502 20:46:14.758927 11089 ts_tablet_manager.cc:1367] T d8d1661cfeee47f8bda913c7a2ca90b0 P a47e5c56f13843018ac584914a8d4707: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 20:46:14.759359 11089 raft_consensus.cc:348] T d8d1661cfeee47f8bda913c7a2ca90b0 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:14.759474 11089 raft_consensus.cc:374] T d8d1661cfeee47f8bda913c7a2ca90b0 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:14.759505 11089 raft_consensus.cc:729] T d8d1661cfeee47f8bda913c7a2ca90b0 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Initialized, Role: FOLLOWER
I20240502 20:46:14.759629 11089 consensus_queue.cc:260] T d8d1661cfeee47f8bda913c7a2ca90b0 P a47e5c56f13843018ac584914a8d4707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:14.759737 11094 tablet_bootstrap.cc:492] T d8d1661cfeee47f8bda913c7a2ca90b0 P 7845c397230d49358ba2fad9db53f97c: Bootstrap starting.
I20240502 20:46:14.759984 11089 ts_tablet_manager.cc:1398] T d8d1661cfeee47f8bda913c7a2ca90b0 P a47e5c56f13843018ac584914a8d4707: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:14.760321 10742 tablet_bootstrap.cc:492] T d8d1661cfeee47f8bda913c7a2ca90b0 P a4bf1df7fa6b4948be502c182bfbb833: No bootstrap required, opened a new log
I20240502 20:46:14.760399 10742 ts_tablet_manager.cc:1367] T d8d1661cfeee47f8bda913c7a2ca90b0 P a4bf1df7fa6b4948be502c182bfbb833: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240502 20:46:14.760609 11094 tablet_bootstrap.cc:654] T d8d1661cfeee47f8bda913c7a2ca90b0 P 7845c397230d49358ba2fad9db53f97c: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:14.760893 4768 raft_consensus.cc:1261] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd [term 2 FOLLOWER]: Refusing update from remote peer a4bf1df7fa6b4948be502c182bfbb833: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 20:46:14.760856 10742 raft_consensus.cc:348] T d8d1661cfeee47f8bda913c7a2ca90b0 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:14.760955 10742 raft_consensus.cc:374] T d8d1661cfeee47f8bda913c7a2ca90b0 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:14.761018 10742 raft_consensus.cc:729] T d8d1661cfeee47f8bda913c7a2ca90b0 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Initialized, Role: FOLLOWER
I20240502 20:46:14.761132 10742 consensus_queue.cc:260] T d8d1661cfeee47f8bda913c7a2ca90b0 P a4bf1df7fa6b4948be502c182bfbb833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:14.761452 10742 ts_tablet_manager.cc:1398] T d8d1661cfeee47f8bda913c7a2ca90b0 P a4bf1df7fa6b4948be502c182bfbb833: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:14.761473 11081 consensus_queue.cc:1035] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:14.762804 11094 tablet_bootstrap.cc:492] T d8d1661cfeee47f8bda913c7a2ca90b0 P 7845c397230d49358ba2fad9db53f97c: No bootstrap required, opened a new log
I20240502 20:46:14.762887 11094 ts_tablet_manager.cc:1367] T d8d1661cfeee47f8bda913c7a2ca90b0 P 7845c397230d49358ba2fad9db53f97c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:46:14.763346 11094 raft_consensus.cc:348] T d8d1661cfeee47f8bda913c7a2ca90b0 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:14.763447 11094 raft_consensus.cc:374] T d8d1661cfeee47f8bda913c7a2ca90b0 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:14.763484 11094 raft_consensus.cc:729] T d8d1661cfeee47f8bda913c7a2ca90b0 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Initialized, Role: FOLLOWER
I20240502 20:46:14.763513 9125 raft_consensus.cc:482] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:14.763592 9125 raft_consensus.cc:504] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:14.763602 11094 consensus_queue.cc:260] T d8d1661cfeee47f8bda913c7a2ca90b0 P 7845c397230d49358ba2fad9db53f97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:14.763852 11094 ts_tablet_manager.cc:1398] T d8d1661cfeee47f8bda913c7a2ca90b0 P 7845c397230d49358ba2fad9db53f97c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:14.763931 6967 raft_consensus.cc:1261] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a47e5c56f13843018ac584914a8d4707 [term 2 FOLLOWER]: Refusing update from remote peer a4bf1df7fa6b4948be502c182bfbb833: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 20:46:14.763971 9125 leader_election.cc:288] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
I20240502 20:46:14.764505 9929 consensus_queue.cc:1035] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:14.764998 7098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77b294bcc497479e8e8bd8e7f9cc1ff2" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201d2b6485084e7dad2cbb722af94779" is_pre_election: true
I20240502 20:46:14.765148 7098 raft_consensus.cc:2454] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 0.
I20240502 20:46:14.765468 4703 leader_election.cc:302] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201d2b6485084e7dad2cbb722af94779, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:46:14.765813 9125 raft_consensus.cc:2781] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:14.765883 9125 raft_consensus.cc:482] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:14.765923 9125 raft_consensus.cc:3037] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:14.766435 6831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77b294bcc497479e8e8bd8e7f9cc1ff2" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833" is_pre_election: true
I20240502 20:46:14.766526 6831 raft_consensus.cc:2454] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 0.
I20240502 20:46:14.767064 9125 raft_consensus.cc:504] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:14.767362 9125 leader_election.cc:288] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 election: Requested vote from peers a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
I20240502 20:46:14.767948 6831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77b294bcc497479e8e8bd8e7f9cc1ff2" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833"
I20240502 20:46:14.768077 6831 raft_consensus.cc:3037] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:14.768672 7098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77b294bcc497479e8e8bd8e7f9cc1ff2" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201d2b6485084e7dad2cbb722af94779"
I20240502 20:46:14.768779 7098 raft_consensus.cc:3037] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:14.769104 6831 raft_consensus.cc:2454] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 1.
I20240502 20:46:14.770097 7098 raft_consensus.cc:2454] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 1.
I20240502 20:46:14.770324 4704 leader_election.cc:302] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4bf1df7fa6b4948be502c182bfbb833, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:46:14.773233 11222 raft_consensus.cc:2781] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:14.773401 11222 raft_consensus.cc:686] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Becoming Leader. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Running, Role: LEADER
I20240502 20:46:14.773465 6831 consensus_queue.cc:237] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } }
I20240502 20:46:14.773653 11222 consensus_queue.cc:237] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:14.776239 6967 raft_consensus.cc:1261] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a47e5c56f13843018ac584914a8d4707 [term 2 FOLLOWER]: Refusing update from remote peer a4bf1df7fa6b4948be502c182bfbb833: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 20:46:14.776366 4507 raft_consensus.cc:1261] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P c3d26eafd51547acb780d4aac5ebe7cc [term 2 FOLLOWER]: Refusing update from remote peer a4bf1df7fa6b4948be502c182bfbb833: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 20:46:14.776698 10776 consensus_queue.cc:1035] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:14.777269 10776 consensus_queue.cc:1035] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:14.777206 4228 catalog_manager.cc:5484] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P ed47c023452a497192f35d2f617bf9bd reported cstate change: term changed from 0 to 1, leader changed from <none> to ed47c023452a497192f35d2f617bf9bd (127.30.32.68). New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } }
I20240502 20:46:14.778928 11081 raft_consensus.cc:2932] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a4bf1df7fa6b4948be502c182bfbb833 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } } }
I20240502 20:46:14.779422 6967 raft_consensus.cc:2932] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a47e5c56f13843018ac584914a8d4707 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } } }
I20240502 20:46:14.780774 4220 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 78d87e2d8c0b4bbd8dd6d7f49cdb4287 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:14.781658 4507 raft_consensus.cc:2932] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P c3d26eafd51547acb780d4aac5ebe7cc [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } } }
I20240502 20:46:14.781749 4228 catalog_manager.cc:5484] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a4bf1df7fa6b4948be502c182bfbb833 reported cstate change: config changed from index 3 to 5, VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) evicted. New cstate: current_term: 2 leader_uuid: "a4bf1df7fa6b4948be502c182bfbb833" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:14.784724 4750 tablet_service.cc:1514] Processing DeleteTablet for tablet 78d87e2d8c0b4bbd8dd6d7f49cdb4287 with delete_type TABLET_DATA_TOMBSTONED (TS ed47c023452a497192f35d2f617bf9bd not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:38526
I20240502 20:46:14.785295 11225 tablet_replica.cc:330] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd: stopping tablet replica
I20240502 20:46:14.785483 11225 raft_consensus.cc:2229] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:14.785614 11225 raft_consensus.cc:2258] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:14.786257 11225 ts_tablet_manager.cc:1875] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:14.788152 11225 ts_tablet_manager.cc:1888] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240502 20:46:14.788220 11225 log.cc:1196] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P ed47c023452a497192f35d2f617bf9bd: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal/wals/78d87e2d8c0b4bbd8dd6d7f49cdb4287
I20240502 20:46:14.788772 4222 catalog_manager.cc:4830] TS ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193): tablet 78d87e2d8c0b4bbd8dd6d7f49cdb4287 (table rebalancer_extra_table_6 [id=f1553e3c7d3c4023865edad13ab86f4a]) successfully deleted
I20240502 20:46:14.801374 11226 ts_tablet_manager.cc:902] T 28da7347f982468281570ffa7503a854 P a47e5c56f13843018ac584914a8d4707: Initiating tablet copy from peer 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:46:14.801901 11226 tablet_copy_client.cc:308] T 28da7347f982468281570ffa7503a854 P a47e5c56f13843018ac584914a8d4707: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.67:42141
I20240502 20:46:14.802639 4658 tablet_copy_service.cc:140] P 923cfab926e7443f8ffd7b21ad0a9d91: Received BeginTabletCopySession request for tablet 28da7347f982468281570ffa7503a854 from peer a47e5c56f13843018ac584914a8d4707 ({username='slave'} at 127.30.32.71:59213)
I20240502 20:46:14.802752 4658 tablet_copy_service.cc:161] P 923cfab926e7443f8ffd7b21ad0a9d91: Beginning new tablet copy session on tablet 28da7347f982468281570ffa7503a854 from peer a47e5c56f13843018ac584914a8d4707 at {username='slave'} at 127.30.32.71:59213: session id = a47e5c56f13843018ac584914a8d4707-28da7347f982468281570ffa7503a854
I20240502 20:46:14.803522 4658 tablet_copy_source_session.cc:205] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:14.804315 11226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28da7347f982468281570ffa7503a854. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:14.806373 11226 tablet_copy_client.cc:783] T 28da7347f982468281570ffa7503a854 P a47e5c56f13843018ac584914a8d4707: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:14.806528 11226 tablet_copy_client.cc:647] T 28da7347f982468281570ffa7503a854 P a47e5c56f13843018ac584914a8d4707: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:14.807126 10270 raft_consensus.cc:1050] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c: attempting to promote NON_VOTER ed47c023452a497192f35d2f617bf9bd to VOTER
I20240502 20:46:14.807510 10270 consensus_queue.cc:237] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } }
I20240502 20:46:14.808750 6967 raft_consensus.cc:1261] T 9258667ccc3e4ef8b26fe75531dd4591 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Refusing update from remote peer 7845c397230d49358ba2fad9db53f97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:14.808813 4768 raft_consensus.cc:1261] T 9258667ccc3e4ef8b26fe75531dd4591 P ed47c023452a497192f35d2f617bf9bd [term 1 LEARNER]: Refusing update from remote peer 7845c397230d49358ba2fad9db53f97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:14.809020 4374 raft_consensus.cc:1261] T 9258667ccc3e4ef8b26fe75531dd4591 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer 7845c397230d49358ba2fad9db53f97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:14.809252 10504 consensus_queue.cc:1035] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:46:14.809458 11080 consensus_queue.cc:1035] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:46:14.810570 11226 tablet_copy_client.cc:523] T 28da7347f982468281570ffa7503a854 P a47e5c56f13843018ac584914a8d4707: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:14.810564 11080 consensus_queue.cc:1035] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:46:14.811918 10504 raft_consensus.cc:2932] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ed47c023452a497192f35d2f617bf9bd (127.30.32.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } } }
I20240502 20:46:14.812377 4768 raft_consensus.cc:2932] T 9258667ccc3e4ef8b26fe75531dd4591 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ed47c023452a497192f35d2f617bf9bd (127.30.32.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } } }
I20240502 20:46:14.813817 6967 raft_consensus.cc:2932] T 9258667ccc3e4ef8b26fe75531dd4591 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ed47c023452a497192f35d2f617bf9bd (127.30.32.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } } }
I20240502 20:46:14.815191 4228 catalog_manager.cc:5484] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c reported cstate change: config changed from index 2 to 3, ed47c023452a497192f35d2f617bf9bd (127.30.32.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7845c397230d49358ba2fad9db53f97c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:14.815375 4374 raft_consensus.cc:2932] T 9258667ccc3e4ef8b26fe75531dd4591 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ed47c023452a497192f35d2f617bf9bd (127.30.32.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } } }
I20240502 20:46:14.817879 11226 tablet_bootstrap.cc:492] T 28da7347f982468281570ffa7503a854 P a47e5c56f13843018ac584914a8d4707: Bootstrap starting.
I20240502 20:46:14.838655 11226 tablet_bootstrap.cc:492] T 28da7347f982468281570ffa7503a854 P a47e5c56f13843018ac584914a8d4707: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:14.839002 11226 tablet_bootstrap.cc:492] T 28da7347f982468281570ffa7503a854 P a47e5c56f13843018ac584914a8d4707: Bootstrap complete.
I20240502 20:46:14.839188 11226 ts_tablet_manager.cc:1367] T 28da7347f982468281570ffa7503a854 P a47e5c56f13843018ac584914a8d4707: Time spent bootstrapping tablet: real 0.021s user 0.018s sys 0.000s
I20240502 20:46:14.839629 11226 raft_consensus.cc:348] T 28da7347f982468281570ffa7503a854 P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } }
I20240502 20:46:14.839738 11226 raft_consensus.cc:729] T 28da7347f982468281570ffa7503a854 P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Initialized, Role: LEARNER
I20240502 20:46:14.839833 11226 consensus_queue.cc:260] T 28da7347f982468281570ffa7503a854 P a47e5c56f13843018ac584914a8d4707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } }
I20240502 20:46:14.840106 4637 raft_consensus.cc:1203] T 51232fccd98c461fb3d9016081f52ba5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 20:46:14.840775 11226 ts_tablet_manager.cc:1398] T 28da7347f982468281570ffa7503a854 P a47e5c56f13843018ac584914a8d4707: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 20:46:14.842039 4658 tablet_copy_service.cc:342] P 923cfab926e7443f8ffd7b21ad0a9d91: Request end of tablet copy session a47e5c56f13843018ac584914a8d4707-28da7347f982468281570ffa7503a854 received from {username='slave'} at 127.30.32.71:59213
I20240502 20:46:14.842123 4658 tablet_copy_service.cc:433] P 923cfab926e7443f8ffd7b21ad0a9d91: ending tablet copy session a47e5c56f13843018ac584914a8d4707-28da7347f982468281570ffa7503a854 on tablet 28da7347f982468281570ffa7503a854 with peer a47e5c56f13843018ac584914a8d4707
I20240502 20:46:14.900892 10994 raft_consensus.cc:1050] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91: attempting to promote NON_VOTER a47e5c56f13843018ac584914a8d4707 to VOTER
I20240502 20:46:14.901367 10994 consensus_queue.cc:237] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } }
I20240502 20:46:14.902963 6967 raft_consensus.cc:1261] T 73efeba3d3094ef387e4efb03cff9d4e P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:14.902976 4374 raft_consensus.cc:1261] T 73efeba3d3094ef387e4efb03cff9d4e P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:14.902983 4768 raft_consensus.cc:1261] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:14.903374 9041 consensus_queue.cc:1035] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:14.903533 10994 consensus_queue.cc:1035] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:14.903712 11238 consensus_queue.cc:1035] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:14.905534 10994 raft_consensus.cc:2932] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:14.905969 6967 raft_consensus.cc:2932] T 73efeba3d3094ef387e4efb03cff9d4e P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:14.905994 4768 raft_consensus.cc:2932] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:14.907536 4374 raft_consensus.cc:2932] T 73efeba3d3094ef387e4efb03cff9d4e P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:14.909512 4227 catalog_manager.cc:5484] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index 4 to 5, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:14.959103 4637 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "73efeba3d3094ef387e4efb03cff9d4e"
dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33928
I20240502 20:46:14.959245 4637 raft_consensus.cc:595] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Received request to transfer leadership
I20240502 20:46:14.969228 6702 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9258667ccc3e4ef8b26fe75531dd4591"
dest_uuid: "7845c397230d49358ba2fad9db53f97c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52252
I20240502 20:46:14.969373 6702 raft_consensus.cc:595] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c [term 1 LEADER]: Received request to transfer leadership
I20240502 20:46:14.979321 4507 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d37730b109d04b6395f75c7552524877"
dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42914
I20240502 20:46:14.979482 4507 raft_consensus.cc:595] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Received request to transfer leadership
I20240502 20:46:15.017309 10138 raft_consensus.cc:482] T d8d1661cfeee47f8bda913c7a2ca90b0 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:15.017436 10138 raft_consensus.cc:504] T d8d1661cfeee47f8bda913c7a2ca90b0 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:15.017798 10138 leader_election.cc:288] T d8d1661cfeee47f8bda913c7a2ca90b0 P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867)
I20240502 20:46:15.018193 6831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d1661cfeee47f8bda913c7a2ca90b0" candidate_uuid: "a47e5c56f13843018ac584914a8d4707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833" is_pre_election: true
I20240502 20:46:15.018368 6831 raft_consensus.cc:2454] T d8d1661cfeee47f8bda913c7a2ca90b0 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a47e5c56f13843018ac584914a8d4707 in term 0.
I20240502 20:46:15.018379 6702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d1661cfeee47f8bda913c7a2ca90b0" candidate_uuid: "a47e5c56f13843018ac584914a8d4707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7845c397230d49358ba2fad9db53f97c" is_pre_election: true
I20240502 20:46:15.018496 6702 raft_consensus.cc:2454] T d8d1661cfeee47f8bda913c7a2ca90b0 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a47e5c56f13843018ac584914a8d4707 in term 0.
I20240502 20:46:15.018812 6902 leader_election.cc:302] T d8d1661cfeee47f8bda913c7a2ca90b0 P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7845c397230d49358ba2fad9db53f97c, a47e5c56f13843018ac584914a8d4707; no voters:
I20240502 20:46:15.019024 10138 raft_consensus.cc:2781] T d8d1661cfeee47f8bda913c7a2ca90b0 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:15.019095 10138 raft_consensus.cc:482] T d8d1661cfeee47f8bda913c7a2ca90b0 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:15.019151 10138 raft_consensus.cc:3037] T d8d1661cfeee47f8bda913c7a2ca90b0 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:15.020359 10138 raft_consensus.cc:504] T d8d1661cfeee47f8bda913c7a2ca90b0 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:15.020490 4637 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "73efeba3d3094ef387e4efb03cff9d4e"
dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33962
I20240502 20:46:15.020609 4637 raft_consensus.cc:595] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Received request to transfer leadership
I20240502 20:46:15.020740 10138 leader_election.cc:288] T d8d1661cfeee47f8bda913c7a2ca90b0 P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 1 election: Requested vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867)
I20240502 20:46:15.021175 6702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d1661cfeee47f8bda913c7a2ca90b0" candidate_uuid: "a47e5c56f13843018ac584914a8d4707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7845c397230d49358ba2fad9db53f97c"
I20240502 20:46:15.021320 6702 raft_consensus.cc:3037] T d8d1661cfeee47f8bda913c7a2ca90b0 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:15.021384 6831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d1661cfeee47f8bda913c7a2ca90b0" candidate_uuid: "a47e5c56f13843018ac584914a8d4707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833"
I20240502 20:46:15.021500 6831 raft_consensus.cc:3037] T d8d1661cfeee47f8bda913c7a2ca90b0 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:15.022346 6702 raft_consensus.cc:2454] T d8d1661cfeee47f8bda913c7a2ca90b0 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a47e5c56f13843018ac584914a8d4707 in term 1.
I20240502 20:46:15.022516 6831 raft_consensus.cc:2454] T d8d1661cfeee47f8bda913c7a2ca90b0 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a47e5c56f13843018ac584914a8d4707 in term 1.
I20240502 20:46:15.022694 6902 leader_election.cc:302] T d8d1661cfeee47f8bda913c7a2ca90b0 P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7845c397230d49358ba2fad9db53f97c, a47e5c56f13843018ac584914a8d4707; no voters:
I20240502 20:46:15.022965 10138 raft_consensus.cc:2781] T d8d1661cfeee47f8bda913c7a2ca90b0 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:15.023129 10138 raft_consensus.cc:686] T d8d1661cfeee47f8bda913c7a2ca90b0 P a47e5c56f13843018ac584914a8d4707 [term 1 LEADER]: Becoming Leader. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Running, Role: LEADER
I20240502 20:46:15.023285 10138 consensus_queue.cc:237] T d8d1661cfeee47f8bda913c7a2ca90b0 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:15.025647 4228 catalog_manager.cc:5484] T d8d1661cfeee47f8bda913c7a2ca90b0 P a47e5c56f13843018ac584914a8d4707 reported cstate change: term changed from 0 to 1, leader changed from <none> to a47e5c56f13843018ac584914a8d4707 (127.30.32.71). New cstate: current_term: 1 leader_uuid: "a47e5c56f13843018ac584914a8d4707" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:15.036782 6702 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9258667ccc3e4ef8b26fe75531dd4591"
dest_uuid: "7845c397230d49358ba2fad9db53f97c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52272
I20240502 20:46:15.036916 6702 raft_consensus.cc:595] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c [term 1 LEADER]: Received request to transfer leadership
I20240502 20:46:15.049175 4507 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d37730b109d04b6395f75c7552524877"
dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42932
I20240502 20:46:15.049322 4507 raft_consensus.cc:595] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Received request to transfer leadership
I20240502 20:46:15.088107 10504 consensus_queue.cc:1035] T 6ff3c95ccfbf486fbc4bfc636afa829b P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:15.092684 9105 consensus_queue.cc:1035] T 6ff3c95ccfbf486fbc4bfc636afa829b P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:15.096549 6702 raft_consensus.cc:1261] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c [term 2 FOLLOWER]: Refusing update from remote peer a4bf1df7fa6b4948be502c182bfbb833: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 20:46:15.096926 9929 consensus_queue.cc:1035] T a9afb75bfc114f59850eaea084a306f0 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:15.130007 10917 raft_consensus.cc:979] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc: : Instructing follower a4bf1df7fa6b4948be502c182bfbb833 to start an election
I20240502 20:46:15.130120 10917 raft_consensus.cc:1067] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Signalling peer a4bf1df7fa6b4948be502c182bfbb833 to start an election
I20240502 20:46:15.130563 6831 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d37730b109d04b6395f75c7552524877"
dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833"
from {username='slave'} at 127.30.32.66:55279
I20240502 20:46:15.130704 6831 raft_consensus.cc:482] T d37730b109d04b6395f75c7552524877 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:46:15.130764 6831 raft_consensus.cc:3037] T d37730b109d04b6395f75c7552524877 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:15.131625 6831 raft_consensus.cc:504] T d37730b109d04b6395f75c7552524877 P a4bf1df7fa6b4948be502c182bfbb833 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } }
I20240502 20:46:15.132090 6831 leader_election.cc:288] T d37730b109d04b6395f75c7552524877 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 election: Requested vote from peers 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141), c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
I20240502 20:46:15.132489 4637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37730b109d04b6395f75c7552524877" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91"
I20240502 20:46:15.132656 4637 raft_consensus.cc:3037] T d37730b109d04b6395f75c7552524877 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:15.132673 7099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37730b109d04b6395f75c7552524877" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "201d2b6485084e7dad2cbb722af94779"
I20240502 20:46:15.132787 7099 raft_consensus.cc:3037] T d37730b109d04b6395f75c7552524877 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:15.132859 4507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37730b109d04b6395f75c7552524877" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc"
I20240502 20:46:15.132968 4507 raft_consensus.cc:3032] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:46:15.133033 4507 raft_consensus.cc:729] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Running, Role: LEADER
I20240502 20:46:15.133178 4507 consensus_queue.cc:260] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } }
I20240502 20:46:15.133383 4507 raft_consensus.cc:3037] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:15.133669 4637 raft_consensus.cc:2454] T d37730b109d04b6395f75c7552524877 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 2.
I20240502 20:46:15.133868 7099 raft_consensus.cc:2454] T d37730b109d04b6395f75c7552524877 P 201d2b6485084e7dad2cbb722af94779 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 2.
I20240502 20:46:15.134196 6768 leader_election.cc:302] T d37730b109d04b6395f75c7552524877 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 201d2b6485084e7dad2cbb722af94779, 923cfab926e7443f8ffd7b21ad0a9d91, a4bf1df7fa6b4948be502c182bfbb833; no voters:
I20240502 20:46:15.134399 9929 raft_consensus.cc:2781] T d37730b109d04b6395f75c7552524877 P a4bf1df7fa6b4948be502c182bfbb833 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:46:15.134436 4507 raft_consensus.cc:2454] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 2.
I20240502 20:46:15.134513 9929 raft_consensus.cc:686] T d37730b109d04b6395f75c7552524877 P a4bf1df7fa6b4948be502c182bfbb833 [term 2 LEADER]: Becoming Leader. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Running, Role: LEADER
I20240502 20:46:15.134668 9929 consensus_queue.cc:237] T d37730b109d04b6395f75c7552524877 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } }
I20240502 20:46:15.136730 4228 catalog_manager.cc:5484] T d37730b109d04b6395f75c7552524877 P a4bf1df7fa6b4948be502c182bfbb833 reported cstate change: term changed from 1 to 2, leader changed from c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) to a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70). New cstate: current_term: 2 leader_uuid: "a4bf1df7fa6b4948be502c182bfbb833" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:15.153690 6967 raft_consensus.cc:1261] T a9afb75bfc114f59850eaea084a306f0 P a47e5c56f13843018ac584914a8d4707 [term 2 FOLLOWER]: Refusing update from remote peer a4bf1df7fa6b4948be502c182bfbb833: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 20:46:15.154129 10776 consensus_queue.cc:1035] T a9afb75bfc114f59850eaea084a306f0 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:15.157243 4768 raft_consensus.cc:1261] T a9afb75bfc114f59850eaea084a306f0 P ed47c023452a497192f35d2f617bf9bd [term 2 FOLLOWER]: Refusing update from remote peer a4bf1df7fa6b4948be502c182bfbb833: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 20:46:15.158551 10776 consensus_queue.cc:1035] T a9afb75bfc114f59850eaea084a306f0 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:15.161195 4228 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 20:46:15.165091 6831 raft_consensus.cc:1261] T d8d1661cfeee47f8bda913c7a2ca90b0 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Refusing update from remote peer a47e5c56f13843018ac584914a8d4707: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:15.165277 6702 raft_consensus.cc:1261] T d8d1661cfeee47f8bda913c7a2ca90b0 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Refusing update from remote peer a47e5c56f13843018ac584914a8d4707: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:15.165542 9156 consensus_queue.cc:1035] T d8d1661cfeee47f8bda913c7a2ca90b0 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:15.165721 11322 consensus_queue.cc:1035] T d8d1661cfeee47f8bda913c7a2ca90b0 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:15.168277 4235 catalog_manager.cc:5576] TS a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867) does not have the latest schema for tablet d8d1661cfeee47f8bda913c7a2ca90b0 (table rebalancer_dynamic_table [id=12a6e673f40f4d7e8dc0931e9521302c]). Expected version 1 got 0
I20240502 20:46:15.169314 4230 catalog_manager.cc:5576] TS 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429) does not have the latest schema for tablet d8d1661cfeee47f8bda913c7a2ca90b0 (table rebalancer_dynamic_table [id=12a6e673f40f4d7e8dc0931e9521302c]). Expected version 1 got 0
I20240502 20:46:15.169788 11356 tablet.cc:1722] T d8d1661cfeee47f8bda913c7a2ca90b0 P a4bf1df7fa6b4948be502c182bfbb833: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 20:46:15.170315 11356 tablet.cc:1620] T d8d1661cfeee47f8bda913c7a2ca90b0 P a4bf1df7fa6b4948be502c182bfbb833: MemRowSet was empty: no flush needed.
I20240502 20:46:15.170392 11355 tablet.cc:1722] T d8d1661cfeee47f8bda913c7a2ca90b0 P a47e5c56f13843018ac584914a8d4707: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 20:46:15.170727 11355 tablet.cc:1620] T d8d1661cfeee47f8bda913c7a2ca90b0 P a47e5c56f13843018ac584914a8d4707: MemRowSet was empty: no flush needed.
I20240502 20:46:15.172806 11357 tablet.cc:1722] T d8d1661cfeee47f8bda913c7a2ca90b0 P 7845c397230d49358ba2fad9db53f97c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 20:46:15.173112 11357 tablet.cc:1620] T d8d1661cfeee47f8bda913c7a2ca90b0 P 7845c397230d49358ba2fad9db53f97c: MemRowSet was empty: no flush needed.
I20240502 20:46:15.172943 4230 catalog_manager.cc:5576] TS 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429) does not have the latest schema for tablet d8d1661cfeee47f8bda913c7a2ca90b0 (table rebalancer_dynamic_table [id=12a6e673f40f4d7e8dc0931e9521302c]). Expected version 1 got 0
I20240502 20:46:15.174021 4235 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: rebalancer_dynamic_table [id=12a6e673f40f4d7e8dc0931e9521302c] alter complete (version 1)
I20240502 20:46:15.174430 4228 catalog_manager.cc:5576] TS a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841) does not have the latest schema for tablet d8d1661cfeee47f8bda913c7a2ca90b0 (table rebalancer_dynamic_table [id=12a6e673f40f4d7e8dc0931e9521302c]). Expected version 1 got 0
I20240502 20:46:15.178758 9929 consensus_queue.cc:1035] T 6201083694c344c8ad63720be89a3cdb P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:15.181646 11081 consensus_queue.cc:1035] T 6201083694c344c8ad63720be89a3cdb P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:15.190868 6957 raft_consensus.cc:1203] T 28da7347f982468281570ffa7503a854 P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 20:46:15.196943 9105 raft_consensus.cc:979] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c: : Instructing follower a47e5c56f13843018ac584914a8d4707 to start an election
I20240502 20:46:15.197113 10504 raft_consensus.cc:1067] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c [term 1 LEADER]: Signalling peer a47e5c56f13843018ac584914a8d4707 to start an election
I20240502 20:46:15.197790 6957 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9258667ccc3e4ef8b26fe75531dd4591"
dest_uuid: "a47e5c56f13843018ac584914a8d4707"
from {username='slave'} at 127.30.32.69:42189
I20240502 20:46:15.197955 6957 raft_consensus.cc:482] T 9258667ccc3e4ef8b26fe75531dd4591 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:46:15.198022 6957 raft_consensus.cc:3037] T 9258667ccc3e4ef8b26fe75531dd4591 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:15.198967 6957 raft_consensus.cc:504] T 9258667ccc3e4ef8b26fe75531dd4591 P a47e5c56f13843018ac584914a8d4707 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } }
I20240502 20:46:15.199457 6957 leader_election.cc:288] T 9258667ccc3e4ef8b26fe75531dd4591 P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 2 election: Requested vote from peers c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827), 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:46:15.199837 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9258667ccc3e4ef8b26fe75531dd4591" candidate_uuid: "a47e5c56f13843018ac584914a8d4707" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ed47c023452a497192f35d2f617bf9bd"
I20240502 20:46:15.199859 4374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9258667ccc3e4ef8b26fe75531dd4591" candidate_uuid: "a47e5c56f13843018ac584914a8d4707" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c92e6e46782849479c8a0c6d82276fb2"
I20240502 20:46:15.199966 4374 raft_consensus.cc:3037] T 9258667ccc3e4ef8b26fe75531dd4591 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:15.199967 4768 raft_consensus.cc:3037] T 9258667ccc3e4ef8b26fe75531dd4591 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:15.200964 4374 raft_consensus.cc:2454] T 9258667ccc3e4ef8b26fe75531dd4591 P c92e6e46782849479c8a0c6d82276fb2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a47e5c56f13843018ac584914a8d4707 in term 2.
I20240502 20:46:15.200964 4768 raft_consensus.cc:2454] T 9258667ccc3e4ef8b26fe75531dd4591 P ed47c023452a497192f35d2f617bf9bd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a47e5c56f13843018ac584914a8d4707 in term 2.
I20240502 20:46:15.201259 6702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9258667ccc3e4ef8b26fe75531dd4591" candidate_uuid: "a47e5c56f13843018ac584914a8d4707" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "7845c397230d49358ba2fad9db53f97c"
I20240502 20:46:15.201375 6702 raft_consensus.cc:3032] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:46:15.201406 6702 raft_consensus.cc:729] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Running, Role: LEADER
I20240502 20:46:15.201557 6702 consensus_queue.cc:260] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } }
I20240502 20:46:15.201733 6702 raft_consensus.cc:3037] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:15.202653 6702 raft_consensus.cc:2454] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a47e5c56f13843018ac584914a8d4707 in term 2.
I20240502 20:46:15.202937 6903 leader_election.cc:302] T 9258667ccc3e4ef8b26fe75531dd4591 P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: a47e5c56f13843018ac584914a8d4707, c92e6e46782849479c8a0c6d82276fb2, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:46:15.203131 10138 raft_consensus.cc:2781] T 9258667ccc3e4ef8b26fe75531dd4591 P a47e5c56f13843018ac584914a8d4707 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:46:15.203207 10138 raft_consensus.cc:686] T 9258667ccc3e4ef8b26fe75531dd4591 P a47e5c56f13843018ac584914a8d4707 [term 2 LEADER]: Becoming Leader. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Running, Role: LEADER
I20240502 20:46:15.203343 10138 consensus_queue.cc:237] T 9258667ccc3e4ef8b26fe75531dd4591 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } }
I20240502 20:46:15.205492 4230 catalog_manager.cc:5484] T 9258667ccc3e4ef8b26fe75531dd4591 P a47e5c56f13843018ac584914a8d4707 reported cstate change: term changed from 1 to 2, leader changed from 7845c397230d49358ba2fad9db53f97c (127.30.32.69) to a47e5c56f13843018ac584914a8d4707 (127.30.32.71). New cstate: current_term: 2 leader_uuid: "a47e5c56f13843018ac584914a8d4707" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:15.208079 11226 ts_tablet_manager.cc:902] T 45ade8c672d8445eaffb179772e3759a P a47e5c56f13843018ac584914a8d4707: Initiating tablet copy from peer c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097)
I20240502 20:46:15.208573 11226 tablet_copy_client.cc:308] T 45ade8c672d8445eaffb179772e3759a P a47e5c56f13843018ac584914a8d4707: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.66:42097
I20240502 20:46:15.209075 4528 tablet_copy_service.cc:140] P c3d26eafd51547acb780d4aac5ebe7cc: Received BeginTabletCopySession request for tablet 45ade8c672d8445eaffb179772e3759a from peer a47e5c56f13843018ac584914a8d4707 ({username='slave'} at 127.30.32.71:46593)
I20240502 20:46:15.209182 4528 tablet_copy_service.cc:161] P c3d26eafd51547acb780d4aac5ebe7cc: Beginning new tablet copy session on tablet 45ade8c672d8445eaffb179772e3759a from peer a47e5c56f13843018ac584914a8d4707 at {username='slave'} at 127.30.32.71:46593: session id = a47e5c56f13843018ac584914a8d4707-45ade8c672d8445eaffb179772e3759a
I20240502 20:46:15.209993 4528 tablet_copy_source_session.cc:205] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:15.210861 11226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45ade8c672d8445eaffb179772e3759a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:15.212960 11226 tablet_copy_client.cc:783] T 45ade8c672d8445eaffb179772e3759a P a47e5c56f13843018ac584914a8d4707: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:15.213137 11226 tablet_copy_client.cc:647] T 45ade8c672d8445eaffb179772e3759a P a47e5c56f13843018ac584914a8d4707: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:15.214326 11226 tablet_copy_client.cc:523] T 45ade8c672d8445eaffb179772e3759a P a47e5c56f13843018ac584914a8d4707: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:15.215462 11226 tablet_bootstrap.cc:492] T 45ade8c672d8445eaffb179772e3759a P a47e5c56f13843018ac584914a8d4707: Bootstrap starting.
I20240502 20:46:15.234535 11226 tablet_bootstrap.cc:492] T 45ade8c672d8445eaffb179772e3759a P a47e5c56f13843018ac584914a8d4707: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:15.234824 11226 tablet_bootstrap.cc:492] T 45ade8c672d8445eaffb179772e3759a P a47e5c56f13843018ac584914a8d4707: Bootstrap complete.
I20240502 20:46:15.235003 11226 ts_tablet_manager.cc:1367] T 45ade8c672d8445eaffb179772e3759a P a47e5c56f13843018ac584914a8d4707: Time spent bootstrapping tablet: real 0.020s user 0.017s sys 0.001s
I20240502 20:46:15.235486 11226 raft_consensus.cc:348] T 45ade8c672d8445eaffb179772e3759a P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } }
I20240502 20:46:15.235613 11226 raft_consensus.cc:729] T 45ade8c672d8445eaffb179772e3759a P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Initialized, Role: LEARNER
I20240502 20:46:15.235713 11226 consensus_queue.cc:260] T 45ade8c672d8445eaffb179772e3759a P a47e5c56f13843018ac584914a8d4707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } }
I20240502 20:46:15.236660 11226 ts_tablet_manager.cc:1398] T 45ade8c672d8445eaffb179772e3759a P a47e5c56f13843018ac584914a8d4707: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 20:46:15.237265 4528 tablet_copy_service.cc:342] P c3d26eafd51547acb780d4aac5ebe7cc: Request end of tablet copy session a47e5c56f13843018ac584914a8d4707-45ade8c672d8445eaffb179772e3759a received from {username='slave'} at 127.30.32.71:46593
I20240502 20:46:15.237370 4528 tablet_copy_service.cc:433] P c3d26eafd51547acb780d4aac5ebe7cc: ending tablet copy session a47e5c56f13843018ac584914a8d4707-45ade8c672d8445eaffb179772e3759a on tablet 45ade8c672d8445eaffb179772e3759a with peer a47e5c56f13843018ac584914a8d4707
I20240502 20:46:15.279419 4230 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 20:46:15.285833 11355 tablet.cc:1722] T d8d1661cfeee47f8bda913c7a2ca90b0 P a47e5c56f13843018ac584914a8d4707: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 20:46:15.286227 11355 tablet.cc:1620] T d8d1661cfeee47f8bda913c7a2ca90b0 P a47e5c56f13843018ac584914a8d4707: MemRowSet was empty: no flush needed.
I20240502 20:46:15.287155 11356 tablet.cc:1722] T d8d1661cfeee47f8bda913c7a2ca90b0 P a4bf1df7fa6b4948be502c182bfbb833: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 20:46:15.287451 11356 tablet.cc:1620] T d8d1661cfeee47f8bda913c7a2ca90b0 P a4bf1df7fa6b4948be502c182bfbb833: MemRowSet was empty: no flush needed.
I20240502 20:46:15.287943 11357 tablet.cc:1722] T d8d1661cfeee47f8bda913c7a2ca90b0 P 7845c397230d49358ba2fad9db53f97c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 20:46:15.288231 11357 tablet.cc:1620] T d8d1661cfeee47f8bda913c7a2ca90b0 P 7845c397230d49358ba2fad9db53f97c: MemRowSet was empty: no flush needed.
I20240502 20:46:15.293143 4230 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: rebalancer_dynamic_table [id=12a6e673f40f4d7e8dc0931e9521302c] alter complete (version 2)
I20240502 20:46:15.293152 4227 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41812:
name: "rebalancer_extra_table_22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:46:15.294956 4637 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "73efeba3d3094ef387e4efb03cff9d4e"
dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34004
I20240502 20:46:15.295086 4637 raft_consensus.cc:595] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Received request to transfer leadership
I20240502 20:46:15.300714 6683 tablet_service.cc:1467] Processing CreateTablet for tablet cfe54b00aa2a4dbe98440c9320a3deef (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=06d19ee6fad04dba931142607aec23bc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:15.301074 6683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfe54b00aa2a4dbe98440c9320a3deef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:15.301355 6681 tablet_service.cc:1467] Processing CreateTablet for tablet 533e7f3c852c4fd9bd5093085a5e8e36 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=06d19ee6fad04dba931142607aec23bc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:15.301616 6681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 533e7f3c852c4fd9bd5093085a5e8e36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:15.303445 11396 tablet_bootstrap.cc:492] T cfe54b00aa2a4dbe98440c9320a3deef P 7845c397230d49358ba2fad9db53f97c: Bootstrap starting.
I20240502 20:46:15.303958 6945 tablet_service.cc:1467] Processing CreateTablet for tablet 533e7f3c852c4fd9bd5093085a5e8e36 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=06d19ee6fad04dba931142607aec23bc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:15.304219 6945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 533e7f3c852c4fd9bd5093085a5e8e36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:15.304234 7078 tablet_service.cc:1467] Processing CreateTablet for tablet cfe54b00aa2a4dbe98440c9320a3deef (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=06d19ee6fad04dba931142607aec23bc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:15.304446 6947 tablet_service.cc:1467] Processing CreateTablet for tablet 3081e646f1b044afac5f8e1e8d45aa6b (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=06d19ee6fad04dba931142607aec23bc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:15.304540 7078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfe54b00aa2a4dbe98440c9320a3deef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:15.304646 6947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3081e646f1b044afac5f8e1e8d45aa6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:15.305330 4618 tablet_service.cc:1467] Processing CreateTablet for tablet cfe54b00aa2a4dbe98440c9320a3deef (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=06d19ee6fad04dba931142607aec23bc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:15.305599 4618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfe54b00aa2a4dbe98440c9320a3deef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:15.306494 11397 tablet_bootstrap.cc:492] T 533e7f3c852c4fd9bd5093085a5e8e36 P a47e5c56f13843018ac584914a8d4707: Bootstrap starting.
I20240502 20:46:15.306731 7077 tablet_service.cc:1467] Processing CreateTablet for tablet 3081e646f1b044afac5f8e1e8d45aa6b (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=06d19ee6fad04dba931142607aec23bc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:15.306968 7077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3081e646f1b044afac5f8e1e8d45aa6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:15.307440 11397 tablet_bootstrap.cc:654] T 533e7f3c852c4fd9bd5093085a5e8e36 P a47e5c56f13843018ac584914a8d4707: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:15.308053 7079 tablet_service.cc:1467] Processing CreateTablet for tablet 533e7f3c852c4fd9bd5093085a5e8e36 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=06d19ee6fad04dba931142607aec23bc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:15.308302 7079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 533e7f3c852c4fd9bd5093085a5e8e36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:15.308360 11396 tablet_bootstrap.cc:654] T cfe54b00aa2a4dbe98440c9320a3deef P 7845c397230d49358ba2fad9db53f97c: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:15.309782 11398 tablet_bootstrap.cc:492] T cfe54b00aa2a4dbe98440c9320a3deef P 201d2b6485084e7dad2cbb722af94779: Bootstrap starting.
I20240502 20:46:15.310350 11397 tablet_bootstrap.cc:492] T 533e7f3c852c4fd9bd5093085a5e8e36 P a47e5c56f13843018ac584914a8d4707: No bootstrap required, opened a new log
I20240502 20:46:15.310431 11397 ts_tablet_manager.cc:1367] T 533e7f3c852c4fd9bd5093085a5e8e36 P a47e5c56f13843018ac584914a8d4707: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 20:46:15.310930 11397 raft_consensus.cc:348] T 533e7f3c852c4fd9bd5093085a5e8e36 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:15.311105 11397 raft_consensus.cc:374] T 533e7f3c852c4fd9bd5093085a5e8e36 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:15.311174 11397 raft_consensus.cc:729] T 533e7f3c852c4fd9bd5093085a5e8e36 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Initialized, Role: FOLLOWER
I20240502 20:46:15.311355 11397 consensus_queue.cc:260] T 533e7f3c852c4fd9bd5093085a5e8e36 P a47e5c56f13843018ac584914a8d4707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:15.311719 11397 ts_tablet_manager.cc:1398] T 533e7f3c852c4fd9bd5093085a5e8e36 P a47e5c56f13843018ac584914a8d4707: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:15.311900 11397 tablet_bootstrap.cc:492] T 3081e646f1b044afac5f8e1e8d45aa6b P a47e5c56f13843018ac584914a8d4707: Bootstrap starting.
I20240502 20:46:15.312899 11397 tablet_bootstrap.cc:654] T 3081e646f1b044afac5f8e1e8d45aa6b P a47e5c56f13843018ac584914a8d4707: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:15.312899 11398 tablet_bootstrap.cc:654] T cfe54b00aa2a4dbe98440c9320a3deef P 201d2b6485084e7dad2cbb722af94779: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:15.313907 9041 raft_consensus.cc:979] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91: : Instructing follower ed47c023452a497192f35d2f617bf9bd to start an election
I20240502 20:46:15.313997 9041 raft_consensus.cc:1067] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Signalling peer ed47c023452a497192f35d2f617bf9bd to start an election
I20240502 20:46:15.314177 11396 tablet_bootstrap.cc:492] T cfe54b00aa2a4dbe98440c9320a3deef P 7845c397230d49358ba2fad9db53f97c: No bootstrap required, opened a new log
I20240502 20:46:15.314275 11396 ts_tablet_manager.cc:1367] T cfe54b00aa2a4dbe98440c9320a3deef P 7845c397230d49358ba2fad9db53f97c: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.001s
I20240502 20:46:15.314538 4768 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "73efeba3d3094ef387e4efb03cff9d4e"
dest_uuid: "ed47c023452a497192f35d2f617bf9bd"
from {username='slave'} at 127.30.32.67:42459
I20240502 20:46:15.314697 4768 raft_consensus.cc:482] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:46:15.314777 4768 raft_consensus.cc:3037] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:15.314851 11396 raft_consensus.cc:348] T cfe54b00aa2a4dbe98440c9320a3deef P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:15.314960 11396 raft_consensus.cc:374] T cfe54b00aa2a4dbe98440c9320a3deef P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:15.314990 11396 raft_consensus.cc:729] T cfe54b00aa2a4dbe98440c9320a3deef P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Initialized, Role: FOLLOWER
I20240502 20:46:15.315095 11396 consensus_queue.cc:260] T cfe54b00aa2a4dbe98440c9320a3deef P 7845c397230d49358ba2fad9db53f97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:15.315356 11396 ts_tablet_manager.cc:1398] T cfe54b00aa2a4dbe98440c9320a3deef P 7845c397230d49358ba2fad9db53f97c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:15.315528 11396 tablet_bootstrap.cc:492] T 533e7f3c852c4fd9bd5093085a5e8e36 P 7845c397230d49358ba2fad9db53f97c: Bootstrap starting.
I20240502 20:46:15.315723 6681 tablet_service.cc:1467] Processing CreateTablet for tablet 3081e646f1b044afac5f8e1e8d45aa6b (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=06d19ee6fad04dba931142607aec23bc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:15.315927 4768 raft_consensus.cc:504] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } }
I20240502 20:46:15.316013 6681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3081e646f1b044afac5f8e1e8d45aa6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:15.316413 4768 leader_election.cc:288] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 2 election: Requested vote from peers 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141), c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
I20240502 20:46:15.316604 11396 tablet_bootstrap.cc:654] T 533e7f3c852c4fd9bd5093085a5e8e36 P 7845c397230d49358ba2fad9db53f97c: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:15.317239 6957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73efeba3d3094ef387e4efb03cff9d4e" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "a47e5c56f13843018ac584914a8d4707"
I20240502 20:46:15.317349 6957 raft_consensus.cc:3037] T 73efeba3d3094ef387e4efb03cff9d4e P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:15.317798 11397 tablet_bootstrap.cc:492] T 3081e646f1b044afac5f8e1e8d45aa6b P a47e5c56f13843018ac584914a8d4707: No bootstrap required, opened a new log
I20240502 20:46:15.317886 11397 ts_tablet_manager.cc:1367] T 3081e646f1b044afac5f8e1e8d45aa6b P a47e5c56f13843018ac584914a8d4707: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 20:46:15.318373 6957 raft_consensus.cc:2454] T 73efeba3d3094ef387e4efb03cff9d4e P a47e5c56f13843018ac584914a8d4707 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 2.
I20240502 20:46:15.318423 11397 raft_consensus.cc:348] T 3081e646f1b044afac5f8e1e8d45aa6b P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:15.318524 11397 raft_consensus.cc:374] T 3081e646f1b044afac5f8e1e8d45aa6b P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:15.318557 11397 raft_consensus.cc:729] T 3081e646f1b044afac5f8e1e8d45aa6b P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Initialized, Role: FOLLOWER
I20240502 20:46:15.318696 11397 consensus_queue.cc:260] T 3081e646f1b044afac5f8e1e8d45aa6b P a47e5c56f13843018ac584914a8d4707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:15.319011 11397 ts_tablet_manager.cc:1398] T 3081e646f1b044afac5f8e1e8d45aa6b P a47e5c56f13843018ac584914a8d4707: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:15.319044 4374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73efeba3d3094ef387e4efb03cff9d4e" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "c92e6e46782849479c8a0c6d82276fb2"
I20240502 20:46:15.319173 4374 raft_consensus.cc:3037] T 73efeba3d3094ef387e4efb03cff9d4e P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:15.319445 11403 tablet_bootstrap.cc:492] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:46:15.320264 4374 raft_consensus.cc:2454] T 73efeba3d3094ef387e4efb03cff9d4e P c92e6e46782849479c8a0c6d82276fb2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 2.
I20240502 20:46:15.320664 4706 leader_election.cc:302] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: a47e5c56f13843018ac584914a8d4707, c92e6e46782849479c8a0c6d82276fb2, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:46:15.320740 11403 tablet_bootstrap.cc:654] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:15.320928 9125 raft_consensus.cc:2781] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:46:15.321152 9125 raft_consensus.cc:686] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd [term 2 LEADER]: Becoming Leader. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Running, Role: LEADER
I20240502 20:46:15.321327 9125 consensus_queue.cc:237] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } }
I20240502 20:46:15.321533 4637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73efeba3d3094ef387e4efb03cff9d4e" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91"
I20240502 20:46:15.321650 4637 raft_consensus.cc:3032] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:46:15.321691 4637 raft_consensus.cc:729] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Running, Role: LEADER
I20240502 20:46:15.321818 4637 consensus_queue.cc:260] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } }
I20240502 20:46:15.322005 4637 raft_consensus.cc:3037] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:15.322513 11403 tablet_bootstrap.cc:492] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91: No bootstrap required, opened a new log
I20240502 20:46:15.322592 11403 ts_tablet_manager.cc:1367] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:46:15.322999 11403 raft_consensus.cc:348] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:15.323293 11403 raft_consensus.cc:374] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:15.323524 11403 raft_consensus.cc:729] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: FOLLOWER
I20240502 20:46:15.323976 11396 tablet_bootstrap.cc:492] T 533e7f3c852c4fd9bd5093085a5e8e36 P 7845c397230d49358ba2fad9db53f97c: No bootstrap required, opened a new log
I20240502 20:46:15.324074 11396 ts_tablet_manager.cc:1367] T 533e7f3c852c4fd9bd5093085a5e8e36 P 7845c397230d49358ba2fad9db53f97c: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 20:46:15.323019 4637 raft_consensus.cc:2454] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 2.
I20240502 20:46:15.324224 11403 consensus_queue.cc:260] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:15.324218 4227 catalog_manager.cc:5484] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd reported cstate change: term changed from 1 to 2, leader changed from 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) to ed47c023452a497192f35d2f617bf9bd (127.30.32.68). New cstate: current_term: 2 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:15.324506 11403 ts_tablet_manager.cc:1398] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:15.324553 11396 raft_consensus.cc:348] T 533e7f3c852c4fd9bd5093085a5e8e36 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:15.324654 11396 raft_consensus.cc:374] T 533e7f3c852c4fd9bd5093085a5e8e36 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:15.324688 11396 raft_consensus.cc:729] T 533e7f3c852c4fd9bd5093085a5e8e36 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Initialized, Role: FOLLOWER
I20240502 20:46:15.324810 11396 consensus_queue.cc:260] T 533e7f3c852c4fd9bd5093085a5e8e36 P 7845c397230d49358ba2fad9db53f97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:15.325142 11396 ts_tablet_manager.cc:1398] T 533e7f3c852c4fd9bd5093085a5e8e36 P 7845c397230d49358ba2fad9db53f97c: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240502 20:46:15.325316 11396 tablet_bootstrap.cc:492] T 3081e646f1b044afac5f8e1e8d45aa6b P 7845c397230d49358ba2fad9db53f97c: Bootstrap starting.
I20240502 20:46:15.326287 11396 tablet_bootstrap.cc:654] T 3081e646f1b044afac5f8e1e8d45aa6b P 7845c397230d49358ba2fad9db53f97c: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:15.327409 11396 tablet_bootstrap.cc:492] T 3081e646f1b044afac5f8e1e8d45aa6b P 7845c397230d49358ba2fad9db53f97c: No bootstrap required, opened a new log
I20240502 20:46:15.327498 11396 ts_tablet_manager.cc:1367] T 3081e646f1b044afac5f8e1e8d45aa6b P 7845c397230d49358ba2fad9db53f97c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:15.327937 11396 raft_consensus.cc:348] T 3081e646f1b044afac5f8e1e8d45aa6b P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:15.328063 11396 raft_consensus.cc:374] T 3081e646f1b044afac5f8e1e8d45aa6b P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:15.328110 11396 raft_consensus.cc:729] T 3081e646f1b044afac5f8e1e8d45aa6b P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Initialized, Role: FOLLOWER
I20240502 20:46:15.328285 11396 consensus_queue.cc:260] T 3081e646f1b044afac5f8e1e8d45aa6b P 7845c397230d49358ba2fad9db53f97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:15.328569 11396 ts_tablet_manager.cc:1398] T 3081e646f1b044afac5f8e1e8d45aa6b P 7845c397230d49358ba2fad9db53f97c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:15.329993 9125 consensus_queue.cc:1035] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:15.332028 11398 tablet_bootstrap.cc:492] T cfe54b00aa2a4dbe98440c9320a3deef P 201d2b6485084e7dad2cbb722af94779: No bootstrap required, opened a new log
I20240502 20:46:15.332120 11398 ts_tablet_manager.cc:1367] T cfe54b00aa2a4dbe98440c9320a3deef P 201d2b6485084e7dad2cbb722af94779: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240502 20:46:15.332877 10922 raft_consensus.cc:482] T cfe54b00aa2a4dbe98440c9320a3deef P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:15.332963 10922 raft_consensus.cc:504] T cfe54b00aa2a4dbe98440c9320a3deef P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:15.333374 10922 leader_election.cc:288] T cfe54b00aa2a4dbe98440c9320a3deef P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643), 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:46:15.333912 7099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfe54b00aa2a4dbe98440c9320a3deef" candidate_uuid: "7845c397230d49358ba2fad9db53f97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201d2b6485084e7dad2cbb722af94779" is_pre_election: true
I20240502 20:46:15.333920 4637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfe54b00aa2a4dbe98440c9320a3deef" candidate_uuid: "7845c397230d49358ba2fad9db53f97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" is_pre_election: true
I20240502 20:46:15.334045 4637 raft_consensus.cc:2454] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7845c397230d49358ba2fad9db53f97c in term 0.
W20240502 20:46:15.334231 6636 leader_election.cc:341] T cfe54b00aa2a4dbe98440c9320a3deef P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Illegal state: must be running to vote when last-logged opid is not known
I20240502 20:46:15.334362 6636 leader_election.cc:302] T cfe54b00aa2a4dbe98440c9320a3deef P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7845c397230d49358ba2fad9db53f97c, 923cfab926e7443f8ffd7b21ad0a9d91; no voters: 201d2b6485084e7dad2cbb722af94779
I20240502 20:46:15.334556 10922 raft_consensus.cc:2781] T cfe54b00aa2a4dbe98440c9320a3deef P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:15.334617 10922 raft_consensus.cc:482] T cfe54b00aa2a4dbe98440c9320a3deef P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:15.334666 10922 raft_consensus.cc:3037] T cfe54b00aa2a4dbe98440c9320a3deef P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:15.335639 10922 raft_consensus.cc:504] T cfe54b00aa2a4dbe98440c9320a3deef P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:15.336063 10922 leader_election.cc:288] T cfe54b00aa2a4dbe98440c9320a3deef P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 election: Requested vote from peers 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643), 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:46:15.336369 11398 raft_consensus.cc:348] T cfe54b00aa2a4dbe98440c9320a3deef P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:15.336540 11398 raft_consensus.cc:374] T cfe54b00aa2a4dbe98440c9320a3deef P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:15.336596 11398 raft_consensus.cc:729] T cfe54b00aa2a4dbe98440c9320a3deef P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Initialized, Role: FOLLOWER
I20240502 20:46:15.336750 11398 consensus_queue.cc:260] T cfe54b00aa2a4dbe98440c9320a3deef P 201d2b6485084e7dad2cbb722af94779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:15.337083 11398 ts_tablet_manager.cc:1398] T cfe54b00aa2a4dbe98440c9320a3deef P 201d2b6485084e7dad2cbb722af94779: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 20:46:15.337217 7098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfe54b00aa2a4dbe98440c9320a3deef" candidate_uuid: "7845c397230d49358ba2fad9db53f97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201d2b6485084e7dad2cbb722af94779"
I20240502 20:46:15.337313 7098 raft_consensus.cc:3037] T cfe54b00aa2a4dbe98440c9320a3deef P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:15.337502 11398 tablet_bootstrap.cc:492] T 3081e646f1b044afac5f8e1e8d45aa6b P 201d2b6485084e7dad2cbb722af94779: Bootstrap starting.
I20240502 20:46:15.337636 4637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfe54b00aa2a4dbe98440c9320a3deef" candidate_uuid: "7845c397230d49358ba2fad9db53f97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91"
I20240502 20:46:15.337757 4637 raft_consensus.cc:3037] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:15.338311 7098 raft_consensus.cc:2454] T cfe54b00aa2a4dbe98440c9320a3deef P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7845c397230d49358ba2fad9db53f97c in term 1.
I20240502 20:46:15.338661 4637 raft_consensus.cc:2454] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7845c397230d49358ba2fad9db53f97c in term 1.
I20240502 20:46:15.339135 11398 tablet_bootstrap.cc:654] T 3081e646f1b044afac5f8e1e8d45aa6b P 201d2b6485084e7dad2cbb722af94779: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:15.339633 6636 leader_election.cc:302] T cfe54b00aa2a4dbe98440c9320a3deef P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201d2b6485084e7dad2cbb722af94779, 7845c397230d49358ba2fad9db53f97c; no voters:
I20240502 20:46:15.339933 10922 raft_consensus.cc:2781] T cfe54b00aa2a4dbe98440c9320a3deef P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:15.340030 10922 raft_consensus.cc:686] T cfe54b00aa2a4dbe98440c9320a3deef P 7845c397230d49358ba2fad9db53f97c [term 1 LEADER]: Becoming Leader. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Running, Role: LEADER
I20240502 20:46:15.340160 10922 consensus_queue.cc:237] T cfe54b00aa2a4dbe98440c9320a3deef P 7845c397230d49358ba2fad9db53f97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:15.340921 11398 tablet_bootstrap.cc:492] T 3081e646f1b044afac5f8e1e8d45aa6b P 201d2b6485084e7dad2cbb722af94779: No bootstrap required, opened a new log
I20240502 20:46:15.341068 11398 ts_tablet_manager.cc:1367] T 3081e646f1b044afac5f8e1e8d45aa6b P 201d2b6485084e7dad2cbb722af94779: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 20:46:15.341917 11398 raft_consensus.cc:348] T 3081e646f1b044afac5f8e1e8d45aa6b P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:15.342077 11398 raft_consensus.cc:374] T 3081e646f1b044afac5f8e1e8d45aa6b P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:15.342150 11398 raft_consensus.cc:729] T 3081e646f1b044afac5f8e1e8d45aa6b P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Initialized, Role: FOLLOWER
I20240502 20:46:15.342339 11398 consensus_queue.cc:260] T 3081e646f1b044afac5f8e1e8d45aa6b P 201d2b6485084e7dad2cbb722af94779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:15.342305 4228 catalog_manager.cc:5484] T cfe54b00aa2a4dbe98440c9320a3deef P 7845c397230d49358ba2fad9db53f97c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7845c397230d49358ba2fad9db53f97c (127.30.32.69). New cstate: current_term: 1 leader_uuid: "7845c397230d49358ba2fad9db53f97c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:15.342808 11398 ts_tablet_manager.cc:1398] T 3081e646f1b044afac5f8e1e8d45aa6b P 201d2b6485084e7dad2cbb722af94779: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:15.343674 9125 consensus_queue.cc:1035] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:15.344874 11398 tablet_bootstrap.cc:492] T 533e7f3c852c4fd9bd5093085a5e8e36 P 201d2b6485084e7dad2cbb722af94779: Bootstrap starting.
I20240502 20:46:15.346196 11398 tablet_bootstrap.cc:654] T 533e7f3c852c4fd9bd5093085a5e8e36 P 201d2b6485084e7dad2cbb722af94779: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:15.347232 11398 tablet_bootstrap.cc:492] T 533e7f3c852c4fd9bd5093085a5e8e36 P 201d2b6485084e7dad2cbb722af94779: No bootstrap required, opened a new log
I20240502 20:46:15.347244 10922 raft_consensus.cc:482] T 533e7f3c852c4fd9bd5093085a5e8e36 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:15.347326 11398 ts_tablet_manager.cc:1367] T 533e7f3c852c4fd9bd5093085a5e8e36 P 201d2b6485084e7dad2cbb722af94779: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:15.347333 10922 raft_consensus.cc:504] T 533e7f3c852c4fd9bd5093085a5e8e36 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:15.347713 10922 leader_election.cc:288] T 533e7f3c852c4fd9bd5093085a5e8e36 P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
I20240502 20:46:15.347775 11398 raft_consensus.cc:348] T 533e7f3c852c4fd9bd5093085a5e8e36 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:15.348122 11398 raft_consensus.cc:374] T 533e7f3c852c4fd9bd5093085a5e8e36 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:15.348130 7098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "533e7f3c852c4fd9bd5093085a5e8e36" candidate_uuid: "7845c397230d49358ba2fad9db53f97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201d2b6485084e7dad2cbb722af94779" is_pre_election: true
I20240502 20:46:15.348237 11398 raft_consensus.cc:729] T 533e7f3c852c4fd9bd5093085a5e8e36 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Initialized, Role: FOLLOWER
I20240502 20:46:15.348242 6957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "533e7f3c852c4fd9bd5093085a5e8e36" candidate_uuid: "7845c397230d49358ba2fad9db53f97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47e5c56f13843018ac584914a8d4707" is_pre_election: true
I20240502 20:46:15.348378 6957 raft_consensus.cc:2454] T 533e7f3c852c4fd9bd5093085a5e8e36 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7845c397230d49358ba2fad9db53f97c in term 0.
I20240502 20:46:15.348487 11398 consensus_queue.cc:260] T 533e7f3c852c4fd9bd5093085a5e8e36 P 201d2b6485084e7dad2cbb722af94779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:15.348702 7098 raft_consensus.cc:2454] T 533e7f3c852c4fd9bd5093085a5e8e36 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7845c397230d49358ba2fad9db53f97c in term 0.
I20240502 20:46:15.348954 11398 ts_tablet_manager.cc:1398] T 533e7f3c852c4fd9bd5093085a5e8e36 P 201d2b6485084e7dad2cbb722af94779: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:15.349052 6639 leader_election.cc:302] T 533e7f3c852c4fd9bd5093085a5e8e36 P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7845c397230d49358ba2fad9db53f97c, a47e5c56f13843018ac584914a8d4707; no voters:
I20240502 20:46:15.349539 10922 raft_consensus.cc:2781] T 533e7f3c852c4fd9bd5093085a5e8e36 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:15.349606 10922 raft_consensus.cc:482] T 533e7f3c852c4fd9bd5093085a5e8e36 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:15.349644 10922 raft_consensus.cc:3037] T 533e7f3c852c4fd9bd5093085a5e8e36 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:15.350744 10922 raft_consensus.cc:504] T 533e7f3c852c4fd9bd5093085a5e8e36 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:15.351121 10922 leader_election.cc:288] T 533e7f3c852c4fd9bd5093085a5e8e36 P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 election: Requested vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
I20240502 20:46:15.351765 6957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "533e7f3c852c4fd9bd5093085a5e8e36" candidate_uuid: "7845c397230d49358ba2fad9db53f97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47e5c56f13843018ac584914a8d4707"
I20240502 20:46:15.351969 7098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "533e7f3c852c4fd9bd5093085a5e8e36" candidate_uuid: "7845c397230d49358ba2fad9db53f97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201d2b6485084e7dad2cbb722af94779"
I20240502 20:46:15.352067 7098 raft_consensus.cc:3037] T 533e7f3c852c4fd9bd5093085a5e8e36 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:15.352312 6957 raft_consensus.cc:3037] T 533e7f3c852c4fd9bd5093085a5e8e36 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:15.353125 7098 raft_consensus.cc:2454] T 533e7f3c852c4fd9bd5093085a5e8e36 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7845c397230d49358ba2fad9db53f97c in term 1.
I20240502 20:46:15.353468 6957 raft_consensus.cc:2454] T 533e7f3c852c4fd9bd5093085a5e8e36 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7845c397230d49358ba2fad9db53f97c in term 1.
I20240502 20:46:15.353819 6639 leader_election.cc:302] T 533e7f3c852c4fd9bd5093085a5e8e36 P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7845c397230d49358ba2fad9db53f97c, a47e5c56f13843018ac584914a8d4707; no voters:
I20240502 20:46:15.354015 10922 raft_consensus.cc:2781] T 533e7f3c852c4fd9bd5093085a5e8e36 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:15.354110 10922 raft_consensus.cc:686] T 533e7f3c852c4fd9bd5093085a5e8e36 P 7845c397230d49358ba2fad9db53f97c [term 1 LEADER]: Becoming Leader. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Running, Role: LEADER
I20240502 20:46:15.354259 10922 consensus_queue.cc:237] T 533e7f3c852c4fd9bd5093085a5e8e36 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:15.356637 4235 catalog_manager.cc:5484] T 533e7f3c852c4fd9bd5093085a5e8e36 P 7845c397230d49358ba2fad9db53f97c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7845c397230d49358ba2fad9db53f97c (127.30.32.69). New cstate: current_term: 1 leader_uuid: "7845c397230d49358ba2fad9db53f97c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:15.396286 4235 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 20:46:15.401989 11355 tablet.cc:1722] T d8d1661cfeee47f8bda913c7a2ca90b0 P a47e5c56f13843018ac584914a8d4707: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 20:46:15.403244 11357 tablet.cc:1722] T d8d1661cfeee47f8bda913c7a2ca90b0 P 7845c397230d49358ba2fad9db53f97c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 20:46:15.405206 11356 tablet.cc:1722] T d8d1661cfeee47f8bda913c7a2ca90b0 P a4bf1df7fa6b4948be502c182bfbb833: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 20:46:15.408102 11425 raft_consensus.cc:1050] T 51232fccd98c461fb3d9016081f52ba5 P 201d2b6485084e7dad2cbb722af94779: attempting to promote NON_VOTER 923cfab926e7443f8ffd7b21ad0a9d91 to VOTER
I20240502 20:46:15.408674 11425 consensus_queue.cc:237] T 51232fccd98c461fb3d9016081f52ba5 P 201d2b6485084e7dad2cbb722af94779 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } }
I20240502 20:46:15.410190 4637 raft_consensus.cc:1261] T 51232fccd98c461fb3d9016081f52ba5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEARNER]: Refusing update from remote peer 201d2b6485084e7dad2cbb722af94779: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:15.410534 4507 raft_consensus.cc:1261] T 51232fccd98c461fb3d9016081f52ba5 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer 201d2b6485084e7dad2cbb722af94779: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:15.410929 10567 consensus_queue.cc:1035] T 51232fccd98c461fb3d9016081f52ba5 P 201d2b6485084e7dad2cbb722af94779 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:46:15.411159 6831 raft_consensus.cc:1261] T 51232fccd98c461fb3d9016081f52ba5 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Refusing update from remote peer 201d2b6485084e7dad2cbb722af94779: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:15.411093 11425 consensus_queue.cc:1035] T 51232fccd98c461fb3d9016081f52ba5 P 201d2b6485084e7dad2cbb722af94779 [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:46:15.412114 11425 consensus_queue.cc:1035] T 51232fccd98c461fb3d9016081f52ba5 P 201d2b6485084e7dad2cbb722af94779 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:46:15.414243 10567 raft_consensus.cc:2932] T 51232fccd98c461fb3d9016081f52ba5 P 201d2b6485084e7dad2cbb722af94779 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:15.415457 4230 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: rebalancer_dynamic_table_new_name [id=12a6e673f40f4d7e8dc0931e9521302c] alter complete (version 3)
I20240502 20:46:15.415814 6831 raft_consensus.cc:2932] T 51232fccd98c461fb3d9016081f52ba5 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:15.417254 4637 raft_consensus.cc:2932] T 51232fccd98c461fb3d9016081f52ba5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:15.417883 4227 catalog_manager.cc:5484] T 51232fccd98c461fb3d9016081f52ba5 P 201d2b6485084e7dad2cbb722af94779 reported cstate change: config changed from index 2 to 3, 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "201d2b6485084e7dad2cbb722af94779" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:15.418296 4507 raft_consensus.cc:2932] T 51232fccd98c461fb3d9016081f52ba5 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:15.421422 7098 consensus_queue.cc:237] T 51232fccd98c461fb3d9016081f52ba5 P 201d2b6485084e7dad2cbb722af94779 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } }
I20240502 20:46:15.422698 4637 raft_consensus.cc:1261] T 51232fccd98c461fb3d9016081f52ba5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer 201d2b6485084e7dad2cbb722af94779: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:15.423090 11427 consensus_queue.cc:1035] T 51232fccd98c461fb3d9016081f52ba5 P 201d2b6485084e7dad2cbb722af94779 [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:15.423447 6831 raft_consensus.cc:1261] T 51232fccd98c461fb3d9016081f52ba5 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Refusing update from remote peer 201d2b6485084e7dad2cbb722af94779: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:15.423825 11428 consensus_queue.cc:1035] T 51232fccd98c461fb3d9016081f52ba5 P 201d2b6485084e7dad2cbb722af94779 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:15.424765 11425 raft_consensus.cc:2932] T 51232fccd98c461fb3d9016081f52ba5 P 201d2b6485084e7dad2cbb722af94779 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:15.425387 4637 raft_consensus.cc:2932] T 51232fccd98c461fb3d9016081f52ba5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:15.425905 6831 raft_consensus.cc:2932] T 51232fccd98c461fb3d9016081f52ba5 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:15.426604 4219 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 51232fccd98c461fb3d9016081f52ba5 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:15.428310 4227 catalog_manager.cc:5484] T 51232fccd98c461fb3d9016081f52ba5 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index 3 to 4, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New cstate: current_term: 1 leader_uuid: "201d2b6485084e7dad2cbb722af94779" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:15.430555 4487 tablet_service.cc:1514] Processing DeleteTablet for tablet 51232fccd98c461fb3d9016081f52ba5 with delete_type TABLET_DATA_TOMBSTONED (TS c3d26eafd51547acb780d4aac5ebe7cc not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48890
I20240502 20:46:15.431008 11432 tablet_replica.cc:330] T 51232fccd98c461fb3d9016081f52ba5 P c3d26eafd51547acb780d4aac5ebe7cc: stopping tablet replica
I20240502 20:46:15.431191 11432 raft_consensus.cc:2229] T 51232fccd98c461fb3d9016081f52ba5 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:15.431347 11432 raft_consensus.cc:2258] T 51232fccd98c461fb3d9016081f52ba5 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:15.431891 11432 ts_tablet_manager.cc:1875] T 51232fccd98c461fb3d9016081f52ba5 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:15.433864 11432 ts_tablet_manager.cc:1888] T 51232fccd98c461fb3d9016081f52ba5 P c3d26eafd51547acb780d4aac5ebe7cc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 20:46:15.433939 11432 log.cc:1196] T 51232fccd98c461fb3d9016081f52ba5 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal/wals/51232fccd98c461fb3d9016081f52ba5
I20240502 20:46:15.434515 4221 catalog_manager.cc:4830] TS c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097): tablet 51232fccd98c461fb3d9016081f52ba5 (table rebalancer_extra_table_12 [id=90e5baaad90e44bc8ebd55fcb5256a71]) successfully deleted
I20240502 20:46:15.489128 9156 raft_consensus.cc:482] T 3081e646f1b044afac5f8e1e8d45aa6b P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:15.489276 9156 raft_consensus.cc:504] T 3081e646f1b044afac5f8e1e8d45aa6b P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:15.489763 9156 leader_election.cc:288] T 3081e646f1b044afac5f8e1e8d45aa6b P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643), 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
I20240502 20:46:15.490089 7098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3081e646f1b044afac5f8e1e8d45aa6b" candidate_uuid: "a47e5c56f13843018ac584914a8d4707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201d2b6485084e7dad2cbb722af94779" is_pre_election: true
I20240502 20:46:15.490233 7098 raft_consensus.cc:2454] T 3081e646f1b044afac5f8e1e8d45aa6b P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a47e5c56f13843018ac584914a8d4707 in term 0.
I20240502 20:46:15.490191 6702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3081e646f1b044afac5f8e1e8d45aa6b" candidate_uuid: "a47e5c56f13843018ac584914a8d4707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7845c397230d49358ba2fad9db53f97c" is_pre_election: true
I20240502 20:46:15.490345 6702 raft_consensus.cc:2454] T 3081e646f1b044afac5f8e1e8d45aa6b P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a47e5c56f13843018ac584914a8d4707 in term 0.
I20240502 20:46:15.490518 6900 leader_election.cc:302] T 3081e646f1b044afac5f8e1e8d45aa6b P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201d2b6485084e7dad2cbb722af94779, a47e5c56f13843018ac584914a8d4707; no voters:
I20240502 20:46:15.490743 9156 raft_consensus.cc:2781] T 3081e646f1b044afac5f8e1e8d45aa6b P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:15.490805 9156 raft_consensus.cc:482] T 3081e646f1b044afac5f8e1e8d45aa6b P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:15.490857 9156 raft_consensus.cc:3037] T 3081e646f1b044afac5f8e1e8d45aa6b P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:15.491879 9156 raft_consensus.cc:504] T 3081e646f1b044afac5f8e1e8d45aa6b P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:15.492271 9156 leader_election.cc:288] T 3081e646f1b044afac5f8e1e8d45aa6b P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 1 election: Requested vote from peers 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643), 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
I20240502 20:46:15.492815 7098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3081e646f1b044afac5f8e1e8d45aa6b" candidate_uuid: "a47e5c56f13843018ac584914a8d4707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201d2b6485084e7dad2cbb722af94779"
I20240502 20:46:15.492961 7098 raft_consensus.cc:3037] T 3081e646f1b044afac5f8e1e8d45aa6b P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:15.493348 6702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3081e646f1b044afac5f8e1e8d45aa6b" candidate_uuid: "a47e5c56f13843018ac584914a8d4707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7845c397230d49358ba2fad9db53f97c"
I20240502 20:46:15.493453 6702 raft_consensus.cc:3037] T 3081e646f1b044afac5f8e1e8d45aa6b P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:15.493860 7098 raft_consensus.cc:2454] T 3081e646f1b044afac5f8e1e8d45aa6b P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a47e5c56f13843018ac584914a8d4707 in term 1.
I20240502 20:46:15.494129 6900 leader_election.cc:302] T 3081e646f1b044afac5f8e1e8d45aa6b P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201d2b6485084e7dad2cbb722af94779, a47e5c56f13843018ac584914a8d4707; no voters:
I20240502 20:46:15.494273 9156 raft_consensus.cc:2781] T 3081e646f1b044afac5f8e1e8d45aa6b P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:15.494330 6702 raft_consensus.cc:2454] T 3081e646f1b044afac5f8e1e8d45aa6b P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a47e5c56f13843018ac584914a8d4707 in term 1.
I20240502 20:46:15.494359 9156 raft_consensus.cc:686] T 3081e646f1b044afac5f8e1e8d45aa6b P a47e5c56f13843018ac584914a8d4707 [term 1 LEADER]: Becoming Leader. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Running, Role: LEADER
I20240502 20:46:15.494498 9156 consensus_queue.cc:237] T 3081e646f1b044afac5f8e1e8d45aa6b P a47e5c56f13843018ac584914a8d4707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:15.496450 4228 catalog_manager.cc:5484] T 3081e646f1b044afac5f8e1e8d45aa6b P a47e5c56f13843018ac584914a8d4707 reported cstate change: term changed from 0 to 1, leader changed from <none> to a47e5c56f13843018ac584914a8d4707 (127.30.32.71). New cstate: current_term: 1 leader_uuid: "a47e5c56f13843018ac584914a8d4707" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:15.518339 4228 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 20:46:15.518517 4228 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 20:46:15.520614 4228 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: Sending DeleteTablet for 3 replicas of tablet d8d1661cfeee47f8bda913c7a2ca90b0
I20240502 20:46:15.521164 6947 tablet_service.cc:1514] Processing DeleteTablet for tablet d8d1661cfeee47f8bda913c7a2ca90b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:15 UTC) from {username='slave'} at 127.0.0.1:45156
I20240502 20:46:15.521252 6681 tablet_service.cc:1514] Processing DeleteTablet for tablet d8d1661cfeee47f8bda913c7a2ca90b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:15 UTC) from {username='slave'} at 127.0.0.1:37506
I20240502 20:46:15.521494 6815 tablet_service.cc:1514] Processing DeleteTablet for tablet d8d1661cfeee47f8bda913c7a2ca90b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:15 UTC) from {username='slave'} at 127.0.0.1:35290
I20240502 20:46:15.521557 11434 tablet_replica.cc:330] T d8d1661cfeee47f8bda913c7a2ca90b0 P a47e5c56f13843018ac584914a8d4707: stopping tablet replica
I20240502 20:46:15.521585 11435 tablet_replica.cc:330] T d8d1661cfeee47f8bda913c7a2ca90b0 P 7845c397230d49358ba2fad9db53f97c: stopping tablet replica
I20240502 20:46:15.521711 11434 raft_consensus.cc:2229] T d8d1661cfeee47f8bda913c7a2ca90b0 P a47e5c56f13843018ac584914a8d4707 [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:15.521714 11435 raft_consensus.cc:2229] T d8d1661cfeee47f8bda913c7a2ca90b0 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:15.521847 11435 raft_consensus.cc:2258] T d8d1661cfeee47f8bda913c7a2ca90b0 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:15.522006 11436 tablet_replica.cc:330] T d8d1661cfeee47f8bda913c7a2ca90b0 P a4bf1df7fa6b4948be502c182bfbb833: stopping tablet replica
I20240502 20:46:15.521952 11434 raft_consensus.cc:2258] T d8d1661cfeee47f8bda913c7a2ca90b0 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:15.522181 11436 raft_consensus.cc:2229] T d8d1661cfeee47f8bda913c7a2ca90b0 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:15.522311 11436 raft_consensus.cc:2258] T d8d1661cfeee47f8bda913c7a2ca90b0 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:15.522903 11435 ts_tablet_manager.cc:1875] T d8d1661cfeee47f8bda913c7a2ca90b0 P 7845c397230d49358ba2fad9db53f97c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:15.522948 11436 ts_tablet_manager.cc:1875] T d8d1661cfeee47f8bda913c7a2ca90b0 P a4bf1df7fa6b4948be502c182bfbb833: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:15.523075 11434 ts_tablet_manager.cc:1875] T d8d1661cfeee47f8bda913c7a2ca90b0 P a47e5c56f13843018ac584914a8d4707: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:15.525156 11436 ts_tablet_manager.cc:1888] T d8d1661cfeee47f8bda913c7a2ca90b0 P a4bf1df7fa6b4948be502c182bfbb833: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:15.525156 11435 ts_tablet_manager.cc:1888] T d8d1661cfeee47f8bda913c7a2ca90b0 P 7845c397230d49358ba2fad9db53f97c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:15.525156 11434 ts_tablet_manager.cc:1888] T d8d1661cfeee47f8bda913c7a2ca90b0 P a47e5c56f13843018ac584914a8d4707: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:15.525231 11435 log.cc:1196] T d8d1661cfeee47f8bda913c7a2ca90b0 P 7845c397230d49358ba2fad9db53f97c: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/wal/wals/d8d1661cfeee47f8bda913c7a2ca90b0
I20240502 20:46:15.525247 11434 log.cc:1196] T d8d1661cfeee47f8bda913c7a2ca90b0 P a47e5c56f13843018ac584914a8d4707: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/wal/wals/d8d1661cfeee47f8bda913c7a2ca90b0
I20240502 20:46:15.525300 11436 log.cc:1196] T d8d1661cfeee47f8bda913c7a2ca90b0 P a4bf1df7fa6b4948be502c182bfbb833: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/wal/wals/d8d1661cfeee47f8bda913c7a2ca90b0
I20240502 20:46:15.525509 11434 ts_tablet_manager.cc:1909] T d8d1661cfeee47f8bda913c7a2ca90b0 P a47e5c56f13843018ac584914a8d4707: Deleting consensus metadata
I20240502 20:46:15.525558 11435 ts_tablet_manager.cc:1909] T d8d1661cfeee47f8bda913c7a2ca90b0 P 7845c397230d49358ba2fad9db53f97c: Deleting consensus metadata
I20240502 20:46:15.525619 11436 ts_tablet_manager.cc:1909] T d8d1661cfeee47f8bda913c7a2ca90b0 P a4bf1df7fa6b4948be502c182bfbb833: Deleting consensus metadata
I20240502 20:46:15.526224 4220 catalog_manager.cc:4830] TS a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867): tablet d8d1661cfeee47f8bda913c7a2ca90b0 (table rebalancer_dynamic_table_new_name [id=12a6e673f40f4d7e8dc0931e9521302c]) successfully deleted
I20240502 20:46:15.526335 4221 catalog_manager.cc:4830] TS 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): tablet d8d1661cfeee47f8bda913c7a2ca90b0 (table rebalancer_dynamic_table_new_name [id=12a6e673f40f4d7e8dc0931e9521302c]) successfully deleted
I20240502 20:46:15.526443 4222 catalog_manager.cc:4830] TS a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): tablet d8d1661cfeee47f8bda913c7a2ca90b0 (table rebalancer_dynamic_table_new_name [id=12a6e673f40f4d7e8dc0931e9521302c]) successfully deleted
I20240502 20:46:15.535400 7098 raft_consensus.cc:1261] T d37730b109d04b6395f75c7552524877 P 201d2b6485084e7dad2cbb722af94779 [term 2 FOLLOWER]: Refusing update from remote peer a4bf1df7fa6b4948be502c182bfbb833: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 20:46:15.535794 9929 consensus_queue.cc:1035] T d37730b109d04b6395f75c7552524877 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:15.596837 4507 raft_consensus.cc:1261] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc [term 2 FOLLOWER]: Refusing update from remote peer a4bf1df7fa6b4948be502c182bfbb833: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 20:46:15.597366 10776 consensus_queue.cc:1035] T d37730b109d04b6395f75c7552524877 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:15.599917 4637 raft_consensus.cc:1261] T d37730b109d04b6395f75c7552524877 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 2 FOLLOWER]: Refusing update from remote peer a4bf1df7fa6b4948be502c182bfbb833: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 20:46:15.605741 10776 consensus_queue.cc:1035] T d37730b109d04b6395f75c7552524877 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:15.612272 6831 consensus_queue.cc:237] T a9afb75bfc114f59850eaea084a306f0 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } }
I20240502 20:46:15.613768 4768 raft_consensus.cc:1261] T a9afb75bfc114f59850eaea084a306f0 P ed47c023452a497192f35d2f617bf9bd [term 2 FOLLOWER]: Refusing update from remote peer a4bf1df7fa6b4948be502c182bfbb833: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 20:46:15.614282 10776 consensus_queue.cc:1035] T a9afb75bfc114f59850eaea084a306f0 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:15.614470 6957 raft_consensus.cc:1261] T a9afb75bfc114f59850eaea084a306f0 P a47e5c56f13843018ac584914a8d4707 [term 2 FOLLOWER]: Refusing update from remote peer a4bf1df7fa6b4948be502c182bfbb833: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 20:46:15.614996 10776 consensus_queue.cc:1035] T a9afb75bfc114f59850eaea084a306f0 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:15.616693 11081 raft_consensus.cc:2932] T a9afb75bfc114f59850eaea084a306f0 P a4bf1df7fa6b4948be502c182bfbb833 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } } }
I20240502 20:46:15.617136 4768 raft_consensus.cc:2932] T a9afb75bfc114f59850eaea084a306f0 P ed47c023452a497192f35d2f617bf9bd [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } } }
I20240502 20:46:15.618849 4220 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a9afb75bfc114f59850eaea084a306f0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:15.619477 6957 raft_consensus.cc:2932] T a9afb75bfc114f59850eaea084a306f0 P a47e5c56f13843018ac584914a8d4707 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } } }
I20240502 20:46:15.619673 4228 catalog_manager.cc:5484] T a9afb75bfc114f59850eaea084a306f0 P a4bf1df7fa6b4948be502c182bfbb833 reported cstate change: config changed from index 3 to 5, VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) evicted. New cstate: current_term: 2 leader_uuid: "a4bf1df7fa6b4948be502c182bfbb833" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:15.622498 4230 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41812:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 20:46:15.624548 6681 tablet_service.cc:1514] Processing DeleteTablet for tablet a9afb75bfc114f59850eaea084a306f0 with delete_type TABLET_DATA_TOMBSTONED (TS 7845c397230d49358ba2fad9db53f97c not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:37506
I20240502 20:46:15.624781 11435 tablet_replica.cc:330] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c: stopping tablet replica
I20240502 20:46:15.624922 11435 raft_consensus.cc:2229] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:15.625061 11435 raft_consensus.cc:2258] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:15.626786 11435 ts_tablet_manager.cc:1875] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:15.629567 11435 ts_tablet_manager.cc:1888] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240502 20:46:15.629647 11435 log.cc:1196] T a9afb75bfc114f59850eaea084a306f0 P 7845c397230d49358ba2fad9db53f97c: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/wal/wals/a9afb75bfc114f59850eaea084a306f0
I20240502 20:46:15.630342 4221 catalog_manager.cc:4830] TS 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): tablet a9afb75bfc114f59850eaea084a306f0 (table rebalancer_extra_table_10 [id=c8bf855c4f414063abc8d3c0e2880292]) successfully deleted
I20240502 20:46:15.630386 6947 tablet_service.cc:1467] Processing CreateTablet for tablet 7037d9125d724460aac2482ae1e3dbde (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dd726af14b9f4e538960924b7e78da85]), partition=
I20240502 20:46:15.630491 6681 tablet_service.cc:1467] Processing CreateTablet for tablet 7037d9125d724460aac2482ae1e3dbde (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dd726af14b9f4e538960924b7e78da85]), partition=
I20240502 20:46:15.630640 6947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7037d9125d724460aac2482ae1e3dbde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:15.630728 6681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7037d9125d724460aac2482ae1e3dbde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:15.632423 11397 tablet_bootstrap.cc:492] T 7037d9125d724460aac2482ae1e3dbde P a47e5c56f13843018ac584914a8d4707: Bootstrap starting.
I20240502 20:46:15.632452 11396 tablet_bootstrap.cc:492] T 7037d9125d724460aac2482ae1e3dbde P 7845c397230d49358ba2fad9db53f97c: Bootstrap starting.
I20240502 20:46:15.632705 7079 tablet_service.cc:1467] Processing CreateTablet for tablet 7037d9125d724460aac2482ae1e3dbde (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dd726af14b9f4e538960924b7e78da85]), partition=
I20240502 20:46:15.632952 7079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7037d9125d724460aac2482ae1e3dbde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:15.633425 11397 tablet_bootstrap.cc:654] T 7037d9125d724460aac2482ae1e3dbde P a47e5c56f13843018ac584914a8d4707: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:15.633630 11396 tablet_bootstrap.cc:654] T 7037d9125d724460aac2482ae1e3dbde P 7845c397230d49358ba2fad9db53f97c: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:15.634517 11396 tablet_bootstrap.cc:492] T 7037d9125d724460aac2482ae1e3dbde P 7845c397230d49358ba2fad9db53f97c: No bootstrap required, opened a new log
I20240502 20:46:15.634558 11397 tablet_bootstrap.cc:492] T 7037d9125d724460aac2482ae1e3dbde P a47e5c56f13843018ac584914a8d4707: No bootstrap required, opened a new log
I20240502 20:46:15.634603 11396 ts_tablet_manager.cc:1367] T 7037d9125d724460aac2482ae1e3dbde P 7845c397230d49358ba2fad9db53f97c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 20:46:15.634632 11397 ts_tablet_manager.cc:1367] T 7037d9125d724460aac2482ae1e3dbde P a47e5c56f13843018ac584914a8d4707: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 20:46:15.634972 11396 raft_consensus.cc:348] T 7037d9125d724460aac2482ae1e3dbde P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:15.635051 11396 raft_consensus.cc:374] T 7037d9125d724460aac2482ae1e3dbde P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:15.635071 11396 raft_consensus.cc:729] T 7037d9125d724460aac2482ae1e3dbde P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Initialized, Role: FOLLOWER
I20240502 20:46:15.635107 11397 raft_consensus.cc:348] T 7037d9125d724460aac2482ae1e3dbde P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:15.635177 11396 consensus_queue.cc:260] T 7037d9125d724460aac2482ae1e3dbde P 7845c397230d49358ba2fad9db53f97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:15.635231 11397 raft_consensus.cc:374] T 7037d9125d724460aac2482ae1e3dbde P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:15.635249 11398 tablet_bootstrap.cc:492] T 7037d9125d724460aac2482ae1e3dbde P 201d2b6485084e7dad2cbb722af94779: Bootstrap starting.
I20240502 20:46:15.635267 11397 raft_consensus.cc:729] T 7037d9125d724460aac2482ae1e3dbde P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Initialized, Role: FOLLOWER
I20240502 20:46:15.635385 11397 consensus_queue.cc:260] T 7037d9125d724460aac2482ae1e3dbde P a47e5c56f13843018ac584914a8d4707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:15.635452 11396 ts_tablet_manager.cc:1398] T 7037d9125d724460aac2482ae1e3dbde P 7845c397230d49358ba2fad9db53f97c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:15.636137 11398 tablet_bootstrap.cc:654] T 7037d9125d724460aac2482ae1e3dbde P 201d2b6485084e7dad2cbb722af94779: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:15.636859 11397 ts_tablet_manager.cc:1398] T 7037d9125d724460aac2482ae1e3dbde P a47e5c56f13843018ac584914a8d4707: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 20:46:15.637832 11398 tablet_bootstrap.cc:492] T 7037d9125d724460aac2482ae1e3dbde P 201d2b6485084e7dad2cbb722af94779: No bootstrap required, opened a new log
I20240502 20:46:15.637938 11398 ts_tablet_manager.cc:1367] T 7037d9125d724460aac2482ae1e3dbde P 201d2b6485084e7dad2cbb722af94779: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:46:15.638461 11398 raft_consensus.cc:348] T 7037d9125d724460aac2482ae1e3dbde P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:15.638571 11398 raft_consensus.cc:374] T 7037d9125d724460aac2482ae1e3dbde P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:15.638600 11398 raft_consensus.cc:729] T 7037d9125d724460aac2482ae1e3dbde P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Initialized, Role: FOLLOWER
I20240502 20:46:15.638725 11398 consensus_queue.cc:260] T 7037d9125d724460aac2482ae1e3dbde P 201d2b6485084e7dad2cbb722af94779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:15.638991 11398 ts_tablet_manager.cc:1398] T 7037d9125d724460aac2482ae1e3dbde P 201d2b6485084e7dad2cbb722af94779: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:15.651580 6957 raft_consensus.cc:1203] T 45ade8c672d8445eaffb179772e3759a P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 20:46:15.677846 6702 raft_consensus.cc:1261] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c [term 2 FOLLOWER]: Refusing update from remote peer a47e5c56f13843018ac584914a8d4707: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 20:46:15.678395 9156 consensus_queue.cc:1035] T 9258667ccc3e4ef8b26fe75531dd4591 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:15.747282 9041 raft_consensus.cc:1050] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91: attempting to promote NON_VOTER a47e5c56f13843018ac584914a8d4707 to VOTER
I20240502 20:46:15.747759 9041 consensus_queue.cc:237] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } }
I20240502 20:46:15.749102 6957 raft_consensus.cc:1261] T 28da7347f982468281570ffa7503a854 P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:15.749480 4507 raft_consensus.cc:1261] T 28da7347f982468281570ffa7503a854 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:15.749699 9041 consensus_queue.cc:1035] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:15.749887 11237 consensus_queue.cc:1035] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:15.749922 6835 raft_consensus.cc:1261] T 28da7347f982468281570ffa7503a854 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:15.751014 11237 consensus_queue.cc:1035] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:15.751797 11237 raft_consensus.cc:2932] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:15.752285 6957 raft_consensus.cc:2932] T 28da7347f982468281570ffa7503a854 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:15.753042 4637 raft_consensus.cc:1261] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91 [term 2 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 20:46:15.753679 11416 consensus_queue.cc:1035] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:15.754135 4507 raft_consensus.cc:2932] T 28da7347f982468281570ffa7503a854 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:15.755815 10922 consensus_queue.cc:1035] T cfe54b00aa2a4dbe98440c9320a3deef P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:15.756278 4227 catalog_manager.cc:5484] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index 7 to 8, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:15.759608 10270 consensus_queue.cc:1035] T 533e7f3c852c4fd9bd5093085a5e8e36 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:15.764835 10922 consensus_queue.cc:1035] T cfe54b00aa2a4dbe98440c9320a3deef P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:15.765815 6835 raft_consensus.cc:2932] T 28da7347f982468281570ffa7503a854 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:15.770232 4374 raft_consensus.cc:1261] T 9258667ccc3e4ef8b26fe75531dd4591 P c92e6e46782849479c8a0c6d82276fb2 [term 2 FOLLOWER]: Refusing update from remote peer a47e5c56f13843018ac584914a8d4707: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 20:46:15.770604 10504 consensus_queue.cc:1035] T 533e7f3c852c4fd9bd5093085a5e8e36 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:15.770992 9156 consensus_queue.cc:1035] T 9258667ccc3e4ef8b26fe75531dd4591 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:15.773810 4768 raft_consensus.cc:1261] T 9258667ccc3e4ef8b26fe75531dd4591 P ed47c023452a497192f35d2f617bf9bd [term 2 FOLLOWER]: Refusing update from remote peer a47e5c56f13843018ac584914a8d4707: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 20:46:15.774534 9156 consensus_queue.cc:1035] T 9258667ccc3e4ef8b26fe75531dd4591 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:15.786856 10504 raft_consensus.cc:482] T 7037d9125d724460aac2482ae1e3dbde P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:15.786986 10504 raft_consensus.cc:504] T 7037d9125d724460aac2482ae1e3dbde P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:15.787401 10504 leader_election.cc:288] T 7037d9125d724460aac2482ae1e3dbde P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
I20240502 20:46:15.787465 4374 raft_consensus.cc:1261] T 73efeba3d3094ef387e4efb03cff9d4e P c92e6e46782849479c8a0c6d82276fb2 [term 2 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 20:46:15.787757 7098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7037d9125d724460aac2482ae1e3dbde" candidate_uuid: "7845c397230d49358ba2fad9db53f97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201d2b6485084e7dad2cbb722af94779" is_pre_election: true
I20240502 20:46:15.787892 7098 raft_consensus.cc:2454] T 7037d9125d724460aac2482ae1e3dbde P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7845c397230d49358ba2fad9db53f97c in term 0.
I20240502 20:46:15.787927 6957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7037d9125d724460aac2482ae1e3dbde" candidate_uuid: "7845c397230d49358ba2fad9db53f97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47e5c56f13843018ac584914a8d4707" is_pre_election: true
I20240502 20:46:15.787995 11409 consensus_queue.cc:1035] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:15.788057 6957 raft_consensus.cc:2454] T 7037d9125d724460aac2482ae1e3dbde P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7845c397230d49358ba2fad9db53f97c in term 0.
I20240502 20:46:15.788275 6636 leader_election.cc:302] T 7037d9125d724460aac2482ae1e3dbde P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201d2b6485084e7dad2cbb722af94779, 7845c397230d49358ba2fad9db53f97c; no voters:
I20240502 20:46:15.788427 10504 raft_consensus.cc:2781] T 7037d9125d724460aac2482ae1e3dbde P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:15.788486 10504 raft_consensus.cc:482] T 7037d9125d724460aac2482ae1e3dbde P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:15.788537 10504 raft_consensus.cc:3037] T 7037d9125d724460aac2482ae1e3dbde P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:15.789621 10504 raft_consensus.cc:504] T 7037d9125d724460aac2482ae1e3dbde P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:15.789965 10504 leader_election.cc:288] T 7037d9125d724460aac2482ae1e3dbde P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 election: Requested vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
I20240502 20:46:15.790323 7098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7037d9125d724460aac2482ae1e3dbde" candidate_uuid: "7845c397230d49358ba2fad9db53f97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201d2b6485084e7dad2cbb722af94779"
I20240502 20:46:15.790433 7098 raft_consensus.cc:3037] T 7037d9125d724460aac2482ae1e3dbde P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:15.790558 6957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7037d9125d724460aac2482ae1e3dbde" candidate_uuid: "7845c397230d49358ba2fad9db53f97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47e5c56f13843018ac584914a8d4707"
I20240502 20:46:15.790665 6957 raft_consensus.cc:3037] T 7037d9125d724460aac2482ae1e3dbde P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:15.790683 6967 raft_consensus.cc:1261] T 73efeba3d3094ef387e4efb03cff9d4e P a47e5c56f13843018ac584914a8d4707 [term 2 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 20:46:15.791039 9125 consensus_queue.cc:1035] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:15.791466 7098 raft_consensus.cc:2454] T 7037d9125d724460aac2482ae1e3dbde P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7845c397230d49358ba2fad9db53f97c in term 1.
I20240502 20:46:15.791711 6957 raft_consensus.cc:2454] T 7037d9125d724460aac2482ae1e3dbde P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7845c397230d49358ba2fad9db53f97c in term 1.
I20240502 20:46:15.791893 6636 leader_election.cc:302] T 7037d9125d724460aac2482ae1e3dbde P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201d2b6485084e7dad2cbb722af94779, 7845c397230d49358ba2fad9db53f97c; no voters:
I20240502 20:46:15.792157 10504 raft_consensus.cc:2781] T 7037d9125d724460aac2482ae1e3dbde P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:15.792238 10504 raft_consensus.cc:686] T 7037d9125d724460aac2482ae1e3dbde P 7845c397230d49358ba2fad9db53f97c [term 1 LEADER]: Becoming Leader. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Running, Role: LEADER
I20240502 20:46:15.792368 10504 consensus_queue.cc:237] T 7037d9125d724460aac2482ae1e3dbde P 7845c397230d49358ba2fad9db53f97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:15.795661 4228 catalog_manager.cc:5484] T 7037d9125d724460aac2482ae1e3dbde P 7845c397230d49358ba2fad9db53f97c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7845c397230d49358ba2fad9db53f97c (127.30.32.69). New cstate: current_term: 1 leader_uuid: "7845c397230d49358ba2fad9db53f97c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:15.888167 4637 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "28da7347f982468281570ffa7503a854"
dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34040
I20240502 20:46:15.888309 4637 raft_consensus.cc:595] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Received request to transfer leadership
I20240502 20:46:15.933804 4228 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 20:46:15.937973 7098 raft_consensus.cc:1261] T 7037d9125d724460aac2482ae1e3dbde P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Refusing update from remote peer 7845c397230d49358ba2fad9db53f97c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:15.938001 6967 raft_consensus.cc:1261] T 7037d9125d724460aac2482ae1e3dbde P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Refusing update from remote peer 7845c397230d49358ba2fad9db53f97c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:15.938356 10504 consensus_queue.cc:1035] T 7037d9125d724460aac2482ae1e3dbde P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:15.938508 10922 consensus_queue.cc:1035] T 7037d9125d724460aac2482ae1e3dbde P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:15.941483 4228 catalog_manager.cc:5576] TS a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841) does not have the latest schema for tablet 7037d9125d724460aac2482ae1e3dbde (table rebalancer_dynamic_table [id=dd726af14b9f4e538960924b7e78da85]). Expected version 1 got 0
I20240502 20:46:15.942219 11547 tablet.cc:1722] T 7037d9125d724460aac2482ae1e3dbde P 7845c397230d49358ba2fad9db53f97c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 20:46:15.942481 4235 catalog_manager.cc:5576] TS 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643) does not have the latest schema for tablet 7037d9125d724460aac2482ae1e3dbde (table rebalancer_dynamic_table [id=dd726af14b9f4e538960924b7e78da85]). Expected version 1 got 0
I20240502 20:46:15.942514 11547 tablet.cc:1620] T 7037d9125d724460aac2482ae1e3dbde P 7845c397230d49358ba2fad9db53f97c: MemRowSet was empty: no flush needed.
I20240502 20:46:15.944008 11548 tablet.cc:1722] T 7037d9125d724460aac2482ae1e3dbde P a47e5c56f13843018ac584914a8d4707: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 20:46:15.944247 4235 catalog_manager.cc:5576] TS a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841) does not have the latest schema for tablet 7037d9125d724460aac2482ae1e3dbde (table rebalancer_dynamic_table [id=dd726af14b9f4e538960924b7e78da85]). Expected version 1 got 0
I20240502 20:46:15.944340 11548 tablet.cc:1620] T 7037d9125d724460aac2482ae1e3dbde P a47e5c56f13843018ac584914a8d4707: MemRowSet was empty: no flush needed.
I20240502 20:46:15.945365 4235 catalog_manager.cc:5576] TS 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429) does not have the latest schema for tablet 7037d9125d724460aac2482ae1e3dbde (table rebalancer_dynamic_table [id=dd726af14b9f4e538960924b7e78da85]). Expected version 1 got 0
I20240502 20:46:15.947065 11549 tablet.cc:1722] T 7037d9125d724460aac2482ae1e3dbde P 201d2b6485084e7dad2cbb722af94779: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 20:46:15.947366 11549 tablet.cc:1620] T 7037d9125d724460aac2482ae1e3dbde P 201d2b6485084e7dad2cbb722af94779: MemRowSet was empty: no flush needed.
I20240502 20:46:15.948372 4228 catalog_manager.cc:5576] TS 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643) does not have the latest schema for tablet 7037d9125d724460aac2482ae1e3dbde (table rebalancer_dynamic_table [id=dd726af14b9f4e538960924b7e78da85]). Expected version 1 got 0
I20240502 20:46:15.951076 4227 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: rebalancer_dynamic_table [id=dd726af14b9f4e538960924b7e78da85] alter complete (version 1)
I20240502 20:46:16.019114 4235 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41812:
name: "rebalancer_extra_table_24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:46:16.025831 6815 tablet_service.cc:1467] Processing CreateTablet for tablet 75e2e48cd7a4497f9ccaae4f712f3ef0 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=35f2d89835f1449e9c86c34a59095ccf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:16.026134 6815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75e2e48cd7a4497f9ccaae4f712f3ef0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:16.026155 6802 tablet_service.cc:1467] Processing CreateTablet for tablet abe3bf6cb45a438fbd10bd5e78febc32 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=35f2d89835f1449e9c86c34a59095ccf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:16.026376 6802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abe3bf6cb45a438fbd10bd5e78febc32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:16.026464 4354 tablet_service.cc:1467] Processing CreateTablet for tablet cedd1c79256a4f32a73b6a88244e6181 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=35f2d89835f1449e9c86c34a59095ccf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:16.026754 4354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cedd1c79256a4f32a73b6a88244e6181. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:16.029058 11551 tablet_bootstrap.cc:492] T abe3bf6cb45a438fbd10bd5e78febc32 P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap starting.
I20240502 20:46:16.029055 4487 tablet_service.cc:1467] Processing CreateTablet for tablet 75e2e48cd7a4497f9ccaae4f712f3ef0 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=35f2d89835f1449e9c86c34a59095ccf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:16.029305 6683 tablet_service.cc:1467] Processing CreateTablet for tablet cedd1c79256a4f32a73b6a88244e6181 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=35f2d89835f1449e9c86c34a59095ccf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:16.029592 6683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cedd1c79256a4f32a73b6a88244e6181. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:16.030119 11551 tablet_bootstrap.cc:654] T abe3bf6cb45a438fbd10bd5e78febc32 P a4bf1df7fa6b4948be502c182bfbb833: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:16.030447 4487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75e2e48cd7a4497f9ccaae4f712f3ef0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:16.030081 7079 tablet_service.cc:1467] Processing CreateTablet for tablet abe3bf6cb45a438fbd10bd5e78febc32 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=35f2d89835f1449e9c86c34a59095ccf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:16.030797 7079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abe3bf6cb45a438fbd10bd5e78febc32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:16.031894 11396 tablet_bootstrap.cc:492] T cedd1c79256a4f32a73b6a88244e6181 P 7845c397230d49358ba2fad9db53f97c: Bootstrap starting.
I20240502 20:46:16.032582 11398 tablet_bootstrap.cc:492] T abe3bf6cb45a438fbd10bd5e78febc32 P 201d2b6485084e7dad2cbb722af94779: Bootstrap starting.
I20240502 20:46:16.032930 11396 tablet_bootstrap.cc:654] T cedd1c79256a4f32a73b6a88244e6181 P 7845c397230d49358ba2fad9db53f97c: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:16.033069 6681 tablet_service.cc:1467] Processing CreateTablet for tablet 75e2e48cd7a4497f9ccaae4f712f3ef0 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=35f2d89835f1449e9c86c34a59095ccf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:16.033308 6681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75e2e48cd7a4497f9ccaae4f712f3ef0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:16.033583 11398 tablet_bootstrap.cc:654] T abe3bf6cb45a438fbd10bd5e78febc32 P 201d2b6485084e7dad2cbb722af94779: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:16.033773 4750 tablet_service.cc:1467] Processing CreateTablet for tablet abe3bf6cb45a438fbd10bd5e78febc32 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=35f2d89835f1449e9c86c34a59095ccf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:16.033864 11512 consensus_queue.cc:1035] T 3081e646f1b044afac5f8e1e8d45aa6b P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:16.034054 4750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abe3bf6cb45a438fbd10bd5e78febc32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:16.037942 6815 tablet_service.cc:1467] Processing CreateTablet for tablet cedd1c79256a4f32a73b6a88244e6181 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=35f2d89835f1449e9c86c34a59095ccf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:16.038205 6815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cedd1c79256a4f32a73b6a88244e6181. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:16.038962 10138 consensus_queue.cc:1035] T 3081e646f1b044afac5f8e1e8d45aa6b P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:16.039731 11557 tablet_bootstrap.cc:492] T abe3bf6cb45a438fbd10bd5e78febc32 P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:46:16.040999 11557 tablet_bootstrap.cc:654] T abe3bf6cb45a438fbd10bd5e78febc32 P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:16.042176 11398 tablet_bootstrap.cc:492] T abe3bf6cb45a438fbd10bd5e78febc32 P 201d2b6485084e7dad2cbb722af94779: No bootstrap required, opened a new log
I20240502 20:46:16.042279 11398 ts_tablet_manager.cc:1367] T abe3bf6cb45a438fbd10bd5e78febc32 P 201d2b6485084e7dad2cbb722af94779: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20240502 20:46:16.042827 11396 tablet_bootstrap.cc:492] T cedd1c79256a4f32a73b6a88244e6181 P 7845c397230d49358ba2fad9db53f97c: No bootstrap required, opened a new log
I20240502 20:46:16.042786 11398 raft_consensus.cc:348] T abe3bf6cb45a438fbd10bd5e78febc32 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:16.042887 11398 raft_consensus.cc:374] T abe3bf6cb45a438fbd10bd5e78febc32 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:16.042908 11396 ts_tablet_manager.cc:1367] T cedd1c79256a4f32a73b6a88244e6181 P 7845c397230d49358ba2fad9db53f97c: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.001s
I20240502 20:46:16.042922 11398 raft_consensus.cc:729] T abe3bf6cb45a438fbd10bd5e78febc32 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Initialized, Role: FOLLOWER
I20240502 20:46:16.043048 11398 consensus_queue.cc:260] T abe3bf6cb45a438fbd10bd5e78febc32 P 201d2b6485084e7dad2cbb722af94779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:16.043320 11398 ts_tablet_manager.cc:1398] T abe3bf6cb45a438fbd10bd5e78febc32 P 201d2b6485084e7dad2cbb722af94779: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:16.043365 11396 raft_consensus.cc:348] T cedd1c79256a4f32a73b6a88244e6181 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:16.043462 11396 raft_consensus.cc:374] T cedd1c79256a4f32a73b6a88244e6181 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:16.043493 11396 raft_consensus.cc:729] T cedd1c79256a4f32a73b6a88244e6181 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Initialized, Role: FOLLOWER
I20240502 20:46:16.043614 11396 consensus_queue.cc:260] T cedd1c79256a4f32a73b6a88244e6181 P 7845c397230d49358ba2fad9db53f97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:16.043906 11396 ts_tablet_manager.cc:1398] T cedd1c79256a4f32a73b6a88244e6181 P 7845c397230d49358ba2fad9db53f97c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:16.044070 11396 tablet_bootstrap.cc:492] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P 7845c397230d49358ba2fad9db53f97c: Bootstrap starting.
I20240502 20:46:16.045166 11557 tablet_bootstrap.cc:492] T abe3bf6cb45a438fbd10bd5e78febc32 P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:46:16.045262 11557 ts_tablet_manager.cc:1367] T abe3bf6cb45a438fbd10bd5e78febc32 P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 20:46:16.045271 11396 tablet_bootstrap.cc:654] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P 7845c397230d49358ba2fad9db53f97c: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:16.045747 11557 raft_consensus.cc:348] T abe3bf6cb45a438fbd10bd5e78febc32 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:16.045851 11557 raft_consensus.cc:374] T abe3bf6cb45a438fbd10bd5e78febc32 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:16.045886 11557 raft_consensus.cc:729] T abe3bf6cb45a438fbd10bd5e78febc32 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:46:16.046016 11557 consensus_queue.cc:260] T abe3bf6cb45a438fbd10bd5e78febc32 P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:16.046240 11551 tablet_bootstrap.cc:492] T abe3bf6cb45a438fbd10bd5e78febc32 P a4bf1df7fa6b4948be502c182bfbb833: No bootstrap required, opened a new log
I20240502 20:46:16.046314 11557 ts_tablet_manager.cc:1398] T abe3bf6cb45a438fbd10bd5e78febc32 P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:16.046339 11551 ts_tablet_manager.cc:1367] T abe3bf6cb45a438fbd10bd5e78febc32 P a4bf1df7fa6b4948be502c182bfbb833: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240502 20:46:16.046571 11553 tablet_bootstrap.cc:492] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:46:16.046810 11558 tablet_bootstrap.cc:492] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:46:16.047518 11553 tablet_bootstrap.cc:654] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:16.047813 11558 tablet_bootstrap.cc:654] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:16.046942 11551 raft_consensus.cc:348] T abe3bf6cb45a438fbd10bd5e78febc32 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:16.048206 11551 raft_consensus.cc:374] T abe3bf6cb45a438fbd10bd5e78febc32 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:16.048300 11551 raft_consensus.cc:729] T abe3bf6cb45a438fbd10bd5e78febc32 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Initialized, Role: FOLLOWER
I20240502 20:46:16.048462 11551 consensus_queue.cc:260] T abe3bf6cb45a438fbd10bd5e78febc32 P a4bf1df7fa6b4948be502c182bfbb833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:16.048799 11551 ts_tablet_manager.cc:1398] T abe3bf6cb45a438fbd10bd5e78febc32 P a4bf1df7fa6b4948be502c182bfbb833: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:16.049712 11551 tablet_bootstrap.cc:492] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap starting.
I20240502 20:46:16.050077 11553 tablet_bootstrap.cc:492] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:46:16.050172 11553 ts_tablet_manager.cc:1367] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 20:46:16.050263 11558 tablet_bootstrap.cc:492] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:46:16.050354 11558 ts_tablet_manager.cc:1367] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 20:46:16.050765 11551 tablet_bootstrap.cc:654] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P a4bf1df7fa6b4948be502c182bfbb833: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:16.050689 11553 raft_consensus.cc:348] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:16.050810 11553 raft_consensus.cc:374] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:16.050853 11553 raft_consensus.cc:729] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:46:16.050874 11558 raft_consensus.cc:348] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:16.050988 11558 raft_consensus.cc:374] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:16.051023 11558 raft_consensus.cc:729] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:46:16.050982 11553 consensus_queue.cc:260] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:16.051178 11558 consensus_queue.cc:260] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:16.051270 11553 ts_tablet_manager.cc:1398] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:16.051465 11558 ts_tablet_manager.cc:1398] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:16.053332 4228 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 20:46:16.059123 11547 tablet.cc:1722] T 7037d9125d724460aac2482ae1e3dbde P 7845c397230d49358ba2fad9db53f97c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 20:46:16.059401 11547 tablet.cc:1620] T 7037d9125d724460aac2482ae1e3dbde P 7845c397230d49358ba2fad9db53f97c: MemRowSet was empty: no flush needed.
I20240502 20:46:16.059502 11548 tablet.cc:1722] T 7037d9125d724460aac2482ae1e3dbde P a47e5c56f13843018ac584914a8d4707: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 20:46:16.059650 11549 tablet.cc:1722] T 7037d9125d724460aac2482ae1e3dbde P 201d2b6485084e7dad2cbb722af94779: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 20:46:16.059798 11548 tablet.cc:1620] T 7037d9125d724460aac2482ae1e3dbde P a47e5c56f13843018ac584914a8d4707: MemRowSet was empty: no flush needed.
I20240502 20:46:16.059900 11549 tablet.cc:1620] T 7037d9125d724460aac2482ae1e3dbde P 201d2b6485084e7dad2cbb722af94779: MemRowSet was empty: no flush needed.
I20240502 20:46:16.065366 4230 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: rebalancer_dynamic_table [id=dd726af14b9f4e538960924b7e78da85] alter complete (version 2)
I20240502 20:46:16.069337 11396 tablet_bootstrap.cc:492] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P 7845c397230d49358ba2fad9db53f97c: No bootstrap required, opened a new log
I20240502 20:46:16.069455 11396 ts_tablet_manager.cc:1367] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P 7845c397230d49358ba2fad9db53f97c: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.002s
I20240502 20:46:16.069576 11551 tablet_bootstrap.cc:492] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P a4bf1df7fa6b4948be502c182bfbb833: No bootstrap required, opened a new log
I20240502 20:46:16.069680 11551 ts_tablet_manager.cc:1367] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P a4bf1df7fa6b4948be502c182bfbb833: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240502 20:46:16.069962 11396 raft_consensus.cc:348] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:16.070086 11396 raft_consensus.cc:374] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:16.070127 11396 raft_consensus.cc:729] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Initialized, Role: FOLLOWER
I20240502 20:46:16.070276 11396 consensus_queue.cc:260] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P 7845c397230d49358ba2fad9db53f97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:16.070410 10270 raft_consensus.cc:482] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:16.070472 11551 raft_consensus.cc:348] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:16.070513 10270 raft_consensus.cc:504] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:16.070583 11551 raft_consensus.cc:374] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:16.070616 11396 ts_tablet_manager.cc:1398] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P 7845c397230d49358ba2fad9db53f97c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:16.070616 11551 raft_consensus.cc:729] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Initialized, Role: FOLLOWER
I20240502 20:46:16.070766 11551 consensus_queue.cc:260] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P a4bf1df7fa6b4948be502c182bfbb833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:16.070865 10270 leader_election.cc:288] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097), a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867)
I20240502 20:46:16.071028 11551 ts_tablet_manager.cc:1398] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P a4bf1df7fa6b4948be502c182bfbb833: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:16.071189 11551 tablet_bootstrap.cc:492] T cedd1c79256a4f32a73b6a88244e6181 P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap starting.
I20240502 20:46:16.071655 4507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75e2e48cd7a4497f9ccaae4f712f3ef0" candidate_uuid: "7845c397230d49358ba2fad9db53f97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" is_pre_election: true
I20240502 20:46:16.071777 4507 raft_consensus.cc:2454] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7845c397230d49358ba2fad9db53f97c in term 0.
I20240502 20:46:16.072216 11551 tablet_bootstrap.cc:654] T cedd1c79256a4f32a73b6a88244e6181 P a4bf1df7fa6b4948be502c182bfbb833: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:16.073038 6835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75e2e48cd7a4497f9ccaae4f712f3ef0" candidate_uuid: "7845c397230d49358ba2fad9db53f97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833" is_pre_election: true
I20240502 20:46:16.073144 6835 raft_consensus.cc:2454] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7845c397230d49358ba2fad9db53f97c in term 0.
I20240502 20:46:16.073441 11551 tablet_bootstrap.cc:492] T cedd1c79256a4f32a73b6a88244e6181 P a4bf1df7fa6b4948be502c182bfbb833: No bootstrap required, opened a new log
I20240502 20:46:16.073510 11551 ts_tablet_manager.cc:1367] T cedd1c79256a4f32a73b6a88244e6181 P a4bf1df7fa6b4948be502c182bfbb833: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:16.074384 6638 leader_election.cc:302] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7845c397230d49358ba2fad9db53f97c, c3d26eafd51547acb780d4aac5ebe7cc; no voters:
I20240502 20:46:16.074376 11551 raft_consensus.cc:348] T cedd1c79256a4f32a73b6a88244e6181 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:16.074494 11551 raft_consensus.cc:374] T cedd1c79256a4f32a73b6a88244e6181 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:16.074532 11551 raft_consensus.cc:729] T cedd1c79256a4f32a73b6a88244e6181 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Initialized, Role: FOLLOWER
I20240502 20:46:16.074703 11551 consensus_queue.cc:260] T cedd1c79256a4f32a73b6a88244e6181 P a4bf1df7fa6b4948be502c182bfbb833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:16.074980 10270 raft_consensus.cc:2781] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:16.074988 11551 ts_tablet_manager.cc:1398] T cedd1c79256a4f32a73b6a88244e6181 P a4bf1df7fa6b4948be502c182bfbb833: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:16.075034 10270 raft_consensus.cc:482] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:16.075070 10270 raft_consensus.cc:3037] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:16.076063 10270 raft_consensus.cc:504] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:16.076440 10270 leader_election.cc:288] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 election: Requested vote from peers c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097), a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867)
I20240502 20:46:16.077075 4507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75e2e48cd7a4497f9ccaae4f712f3ef0" candidate_uuid: "7845c397230d49358ba2fad9db53f97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc"
I20240502 20:46:16.077191 4507 raft_consensus.cc:3037] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:16.078202 4507 raft_consensus.cc:2454] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7845c397230d49358ba2fad9db53f97c in term 1.
I20240502 20:46:16.078434 6835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75e2e48cd7a4497f9ccaae4f712f3ef0" candidate_uuid: "7845c397230d49358ba2fad9db53f97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833"
I20240502 20:46:16.078526 6835 raft_consensus.cc:3037] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:16.079442 6835 raft_consensus.cc:2454] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7845c397230d49358ba2fad9db53f97c in term 1.
I20240502 20:46:16.079725 6637 leader_election.cc:302] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7845c397230d49358ba2fad9db53f97c, a4bf1df7fa6b4948be502c182bfbb833; no voters:
I20240502 20:46:16.080902 10270 raft_consensus.cc:2781] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:16.081014 10270 raft_consensus.cc:686] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P 7845c397230d49358ba2fad9db53f97c [term 1 LEADER]: Becoming Leader. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Running, Role: LEADER
I20240502 20:46:16.081138 10270 consensus_queue.cc:237] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } }
I20240502 20:46:16.081905 11428 raft_consensus.cc:482] T abe3bf6cb45a438fbd10bd5e78febc32 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:16.082023 11428 raft_consensus.cc:504] T abe3bf6cb45a438fbd10bd5e78febc32 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:16.082470 11428 leader_election.cc:288] T abe3bf6cb45a438fbd10bd5e78febc32 P 201d2b6485084e7dad2cbb722af94779 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867), ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:46:16.083132 6831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abe3bf6cb45a438fbd10bd5e78febc32" candidate_uuid: "201d2b6485084e7dad2cbb722af94779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833" is_pre_election: true
I20240502 20:46:16.083133 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abe3bf6cb45a438fbd10bd5e78febc32" candidate_uuid: "201d2b6485084e7dad2cbb722af94779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd" is_pre_election: true
I20240502 20:46:16.083249 6831 raft_consensus.cc:2454] T abe3bf6cb45a438fbd10bd5e78febc32 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 201d2b6485084e7dad2cbb722af94779 in term 0.
I20240502 20:46:16.083266 4768 raft_consensus.cc:2454] T abe3bf6cb45a438fbd10bd5e78febc32 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 201d2b6485084e7dad2cbb722af94779 in term 0.
I20240502 20:46:16.083544 7033 leader_election.cc:302] T abe3bf6cb45a438fbd10bd5e78febc32 P 201d2b6485084e7dad2cbb722af94779 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201d2b6485084e7dad2cbb722af94779, a4bf1df7fa6b4948be502c182bfbb833; no voters:
I20240502 20:46:16.083851 11428 raft_consensus.cc:2781] T abe3bf6cb45a438fbd10bd5e78febc32 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:16.083906 11428 raft_consensus.cc:482] T abe3bf6cb45a438fbd10bd5e78febc32 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:16.083958 11428 raft_consensus.cc:3037] T abe3bf6cb45a438fbd10bd5e78febc32 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:16.084937 11428 raft_consensus.cc:504] T abe3bf6cb45a438fbd10bd5e78febc32 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:16.085331 11428 leader_election.cc:288] T abe3bf6cb45a438fbd10bd5e78febc32 P 201d2b6485084e7dad2cbb722af94779 [CANDIDATE]: Term 1 election: Requested vote from peers a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867), ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:46:16.085204 4235 catalog_manager.cc:5484] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P 7845c397230d49358ba2fad9db53f97c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7845c397230d49358ba2fad9db53f97c (127.30.32.69). New cstate: current_term: 1 leader_uuid: "7845c397230d49358ba2fad9db53f97c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } health_report { overall_health: HEALTHY } } }
I20240502 20:46:16.086081 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abe3bf6cb45a438fbd10bd5e78febc32" candidate_uuid: "201d2b6485084e7dad2cbb722af94779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd"
I20240502 20:46:16.086207 4768 raft_consensus.cc:3037] T abe3bf6cb45a438fbd10bd5e78febc32 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:16.086537 6835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abe3bf6cb45a438fbd10bd5e78febc32" candidate_uuid: "201d2b6485084e7dad2cbb722af94779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833"
I20240502 20:46:16.086627 6835 raft_consensus.cc:3037] T abe3bf6cb45a438fbd10bd5e78febc32 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:16.087316 4768 raft_consensus.cc:2454] T abe3bf6cb45a438fbd10bd5e78febc32 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 201d2b6485084e7dad2cbb722af94779 in term 1.
I20240502 20:46:16.087647 6835 raft_consensus.cc:2454] T abe3bf6cb45a438fbd10bd5e78febc32 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 201d2b6485084e7dad2cbb722af94779 in term 1.
I20240502 20:46:16.087889 7035 leader_election.cc:302] T abe3bf6cb45a438fbd10bd5e78febc32 P 201d2b6485084e7dad2cbb722af94779 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201d2b6485084e7dad2cbb722af94779, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:46:16.088326 11428 raft_consensus.cc:2781] T abe3bf6cb45a438fbd10bd5e78febc32 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:16.088423 11428 raft_consensus.cc:686] T abe3bf6cb45a438fbd10bd5e78febc32 P 201d2b6485084e7dad2cbb722af94779 [term 1 LEADER]: Becoming Leader. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Running, Role: LEADER
I20240502 20:46:16.088551 11428 consensus_queue.cc:237] T abe3bf6cb45a438fbd10bd5e78febc32 P 201d2b6485084e7dad2cbb722af94779 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:16.090663 4230 catalog_manager.cc:5484] T abe3bf6cb45a438fbd10bd5e78febc32 P 201d2b6485084e7dad2cbb722af94779 reported cstate change: term changed from 0 to 1, leader changed from <none> to 201d2b6485084e7dad2cbb722af94779 (127.30.32.72). New cstate: current_term: 1 leader_uuid: "201d2b6485084e7dad2cbb722af94779" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:16.127744 4637 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "28da7347f982468281570ffa7503a854"
dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34068
I20240502 20:46:16.127910 4637 raft_consensus.cc:595] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Received request to transfer leadership
I20240502 20:46:16.161429 11237 raft_consensus.cc:979] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91: : Instructing follower c3d26eafd51547acb780d4aac5ebe7cc to start an election
I20240502 20:46:16.161531 10917 raft_consensus.cc:1050] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc: attempting to promote NON_VOTER a47e5c56f13843018ac584914a8d4707 to VOTER
I20240502 20:46:16.161582 11237 raft_consensus.cc:1067] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Signalling peer c3d26eafd51547acb780d4aac5ebe7cc to start an election
I20240502 20:46:16.162017 10917 consensus_queue.cc:237] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } }
I20240502 20:46:16.162039 4507 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "28da7347f982468281570ffa7503a854"
dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc"
from {username='slave'} at 127.30.32.67:59037
I20240502 20:46:16.162230 4507 raft_consensus.cc:482] T 28da7347f982468281570ffa7503a854 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:46:16.162313 4507 raft_consensus.cc:3037] T 28da7347f982468281570ffa7503a854 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:16.163444 4507 raft_consensus.cc:504] T 28da7347f982468281570ffa7503a854 P c3d26eafd51547acb780d4aac5ebe7cc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } }
I20240502 20:46:16.163530 6835 raft_consensus.cc:1261] T 45ade8c672d8445eaffb179772e3759a P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:16.163810 4507 leader_election.cc:288] T 28da7347f982468281570ffa7503a854 P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 2 election: Requested vote from peers 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141), a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
I20240502 20:46:16.164106 6835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28da7347f982468281570ffa7503a854" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833"
I20240502 20:46:16.164157 11600 consensus_queue.cc:1035] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:16.164219 6835 raft_consensus.cc:3037] T 28da7347f982468281570ffa7503a854 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:16.164884 6957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28da7347f982468281570ffa7503a854" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "a47e5c56f13843018ac584914a8d4707"
I20240502 20:46:16.165024 6957 raft_consensus.cc:3037] T 28da7347f982468281570ffa7503a854 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:16.165375 6835 raft_consensus.cc:2454] T 28da7347f982468281570ffa7503a854 P a4bf1df7fa6b4948be502c182bfbb833 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 2.
I20240502 20:46:16.166020 6957 raft_consensus.cc:2454] T 28da7347f982468281570ffa7503a854 P a47e5c56f13843018ac584914a8d4707 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 2.
I20240502 20:46:16.166730 6967 raft_consensus.cc:1261] T 45ade8c672d8445eaffb179772e3759a P a47e5c56f13843018ac584914a8d4707 [term 1 LEARNER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:16.167044 4444 leader_election.cc:302] T 28da7347f982468281570ffa7503a854 P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: a47e5c56f13843018ac584914a8d4707, a4bf1df7fa6b4948be502c182bfbb833, c3d26eafd51547acb780d4aac5ebe7cc; no voters:
I20240502 20:46:16.167227 11600 raft_consensus.cc:2781] T 28da7347f982468281570ffa7503a854 P c3d26eafd51547acb780d4aac5ebe7cc [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:46:16.167346 11600 raft_consensus.cc:686] T 28da7347f982468281570ffa7503a854 P c3d26eafd51547acb780d4aac5ebe7cc [term 2 LEADER]: Becoming Leader. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Running, Role: LEADER
I20240502 20:46:16.167376 10873 consensus_queue.cc:1035] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:16.167579 11600 consensus_queue.cc:237] T 28da7347f982468281570ffa7503a854 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } }
I20240502 20:46:16.167963 4637 raft_consensus.cc:1261] T 45ade8c672d8445eaffb179772e3759a P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:16.168239 4638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28da7347f982468281570ffa7503a854" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91"
I20240502 20:46:16.168339 4638 raft_consensus.cc:3032] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:46:16.168376 4638 raft_consensus.cc:729] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Running, Role: LEADER
I20240502 20:46:16.168524 4638 consensus_queue.cc:260] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } }
I20240502 20:46:16.168715 4638 raft_consensus.cc:3037] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:16.169929 4235 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 20:46:16.170025 4230 catalog_manager.cc:5484] T 28da7347f982468281570ffa7503a854 P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: term changed from 1 to 2, leader changed from 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) to c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66). New cstate: current_term: 2 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:16.171279 4638 raft_consensus.cc:2454] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 2.
I20240502 20:46:16.172267 11600 consensus_queue.cc:1035] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:16.174124 11599 raft_consensus.cc:2932] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:16.174213 6967 raft_consensus.cc:2932] T 45ade8c672d8445eaffb179772e3759a P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:16.176529 11547 tablet.cc:1722] T 7037d9125d724460aac2482ae1e3dbde P 7845c397230d49358ba2fad9db53f97c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 20:46:16.176875 6831 raft_consensus.cc:2932] T 45ade8c672d8445eaffb179772e3759a P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:16.177532 11549 tablet.cc:1722] T 7037d9125d724460aac2482ae1e3dbde P 201d2b6485084e7dad2cbb722af94779: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 20:46:16.178521 4638 raft_consensus.cc:2932] T 45ade8c672d8445eaffb179772e3759a P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:16.179400 4235 catalog_manager.cc:5484] T 45ade8c672d8445eaffb179772e3759a P a47e5c56f13843018ac584914a8d4707 reported cstate change: config changed from index 7 to 8, a47e5c56f13843018ac584914a8d4707 (127.30.32.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:16.185797 4768 consensus_queue.cc:237] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } }
I20240502 20:46:16.188233 4374 raft_consensus.cc:1261] T 73efeba3d3094ef387e4efb03cff9d4e P c92e6e46782849479c8a0c6d82276fb2 [term 2 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 20:46:16.188767 9125 consensus_queue.cc:1035] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:16.189265 6957 raft_consensus.cc:1261] T 73efeba3d3094ef387e4efb03cff9d4e P a47e5c56f13843018ac584914a8d4707 [term 2 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 20:46:16.190905 9125 consensus_queue.cc:1035] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:16.192523 4507 consensus_queue.cc:237] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } }
I20240502 20:46:16.193214 9125 raft_consensus.cc:2932] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:16.194753 6831 raft_consensus.cc:1261] T 45ade8c672d8445eaffb179772e3759a P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:46:16.195477 4222 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 73efeba3d3094ef387e4efb03cff9d4e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:16.195827 10873 consensus_queue.cc:1035] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:16.198254 6957 raft_consensus.cc:1261] T 45ade8c672d8445eaffb179772e3759a P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:46:16.198804 4374 raft_consensus.cc:2932] T 73efeba3d3094ef387e4efb03cff9d4e P c92e6e46782849479c8a0c6d82276fb2 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:16.192862 4227 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: rebalancer_dynamic_table_new_name [id=dd726af14b9f4e538960924b7e78da85] alter complete (version 3)
I20240502 20:46:16.200778 6957 raft_consensus.cc:2932] T 73efeba3d3094ef387e4efb03cff9d4e P a47e5c56f13843018ac584914a8d4707 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:16.201020 10873 consensus_queue.cc:1035] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:16.201467 6835 consensus_queue.cc:237] T d37730b109d04b6395f75c7552524877 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } }
I20240502 20:46:16.202155 4227 catalog_manager.cc:5484] T 73efeba3d3094ef387e4efb03cff9d4e P c92e6e46782849479c8a0c6d82276fb2 reported cstate change: config changed from index 5 to 7, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New cstate: current_term: 2 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:16.202991 7099 raft_consensus.cc:1261] T d37730b109d04b6395f75c7552524877 P 201d2b6485084e7dad2cbb722af94779 [term 2 FOLLOWER]: Refusing update from remote peer a4bf1df7fa6b4948be502c182bfbb833: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 20:46:16.203267 4638 raft_consensus.cc:1261] T d37730b109d04b6395f75c7552524877 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 2 FOLLOWER]: Refusing update from remote peer a4bf1df7fa6b4948be502c182bfbb833: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 20:46:16.203776 11081 consensus_queue.cc:1035] T d37730b109d04b6395f75c7552524877 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:46:16.204264 11081 consensus_queue.cc:1035] T d37730b109d04b6395f75c7552524877 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:46:16.206542 11081 raft_consensus.cc:2932] T d37730b109d04b6395f75c7552524877 P a4bf1df7fa6b4948be502c182bfbb833 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:16.207123 4638 raft_consensus.cc:2932] T d37730b109d04b6395f75c7552524877 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:16.208217 11548 tablet.cc:1722] T 7037d9125d724460aac2482ae1e3dbde P a47e5c56f13843018ac584914a8d4707: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 20:46:16.210956 4618 tablet_service.cc:1514] Processing DeleteTablet for tablet 73efeba3d3094ef387e4efb03cff9d4e with delete_type TABLET_DATA_TOMBSTONED (TS 923cfab926e7443f8ffd7b21ad0a9d91 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57068
I20240502 20:46:16.211290 11606 tablet_replica.cc:330] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91: stopping tablet replica
I20240502 20:46:16.211434 11606 raft_consensus.cc:2229] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:16.211535 11606 raft_consensus.cc:2258] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:16.211491 4230 catalog_manager.cc:5484] T d37730b109d04b6395f75c7552524877 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index 8 to 10, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New cstate: current_term: 2 leader_uuid: "a4bf1df7fa6b4948be502c182bfbb833" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:16.212746 11606 ts_tablet_manager.cc:1875] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:16.213444 11600 raft_consensus.cc:2932] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:16.213654 4220 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d37730b109d04b6395f75c7552524877 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:16.213999 6957 raft_consensus.cc:2932] T 45ade8c672d8445eaffb179772e3759a P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:16.215601 11606 ts_tablet_manager.cc:1888] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 20:46:16.215704 11606 log.cc:1196] T 73efeba3d3094ef387e4efb03cff9d4e P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal/wals/73efeba3d3094ef387e4efb03cff9d4e
I20240502 20:46:16.216276 4219 catalog_manager.cc:4830] TS 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141): tablet 73efeba3d3094ef387e4efb03cff9d4e (table rebalance_test_table_15 [id=57f2e4fb3cca4e408bd33dac3f4b9d0c]) successfully deleted
I20240502 20:46:16.212870 7099 raft_consensus.cc:2932] T d37730b109d04b6395f75c7552524877 P 201d2b6485084e7dad2cbb722af94779 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: false } } }
I20240502 20:46:16.218246 4221 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 45ade8c672d8445eaffb179772e3759a with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:16.218833 6831 raft_consensus.cc:2932] T 45ade8c672d8445eaffb179772e3759a P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:16.220387 4228 catalog_manager.cc:5484] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: config changed from index 8 to 9, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New cstate: current_term: 1 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:16.223989 4487 tablet_service.cc:1514] Processing DeleteTablet for tablet d37730b109d04b6395f75c7552524877 with delete_type TABLET_DATA_TOMBSTONED (TS c3d26eafd51547acb780d4aac5ebe7cc not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:48890
I20240502 20:46:16.224617 11607 tablet_replica.cc:330] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc: stopping tablet replica
I20240502 20:46:16.224774 11607 raft_consensus.cc:2229] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:16.224911 11607 raft_consensus.cc:2258] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:16.226720 4618 tablet_service.cc:1514] Processing DeleteTablet for tablet 45ade8c672d8445eaffb179772e3759a with delete_type TABLET_DATA_TOMBSTONED (TS 923cfab926e7443f8ffd7b21ad0a9d91 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:57068
I20240502 20:46:16.227721 11607 ts_tablet_manager.cc:1875] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:16.231036 11606 tablet_replica.cc:330] T 45ade8c672d8445eaffb179772e3759a P 923cfab926e7443f8ffd7b21ad0a9d91: stopping tablet replica
I20240502 20:46:16.231398 11607 ts_tablet_manager.cc:1888] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 20:46:16.231484 11607 log.cc:1196] T d37730b109d04b6395f75c7552524877 P c3d26eafd51547acb780d4aac5ebe7cc: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-1/wal/wals/d37730b109d04b6395f75c7552524877
I20240502 20:46:16.231261 11606 raft_consensus.cc:2229] T 45ade8c672d8445eaffb179772e3759a P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:16.232071 4221 catalog_manager.cc:4830] TS c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097): tablet d37730b109d04b6395f75c7552524877 (table rebalance_test_table_7 [id=1e32a0fff569475cbd44b12c4fd5e8e6]) successfully deleted
I20240502 20:46:16.232311 11606 raft_consensus.cc:2258] T 45ade8c672d8445eaffb179772e3759a P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:16.237814 11606 ts_tablet_manager.cc:1875] T 45ade8c672d8445eaffb179772e3759a P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:16.240677 11606 ts_tablet_manager.cc:1888] T 45ade8c672d8445eaffb179772e3759a P 923cfab926e7443f8ffd7b21ad0a9d91: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 20:46:16.240803 11606 log.cc:1196] T 45ade8c672d8445eaffb179772e3759a P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal/wals/45ade8c672d8445eaffb179772e3759a
I20240502 20:46:16.241466 4219 catalog_manager.cc:4830] TS 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141): tablet 45ade8c672d8445eaffb179772e3759a (table rebalance_test_table_19 [id=46a52eb51c614964a91ee65db96cf854]) successfully deleted
I20240502 20:46:16.260950 6957 consensus_queue.cc:237] T 9258667ccc3e4ef8b26fe75531dd4591 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } }
I20240502 20:46:16.264971 4374 raft_consensus.cc:1261] T 9258667ccc3e4ef8b26fe75531dd4591 P c92e6e46782849479c8a0c6d82276fb2 [term 2 FOLLOWER]: Refusing update from remote peer a47e5c56f13843018ac584914a8d4707: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 20:46:16.265450 10138 consensus_queue.cc:1035] T 9258667ccc3e4ef8b26fe75531dd4591 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:16.266402 4768 raft_consensus.cc:1261] T 9258667ccc3e4ef8b26fe75531dd4591 P ed47c023452a497192f35d2f617bf9bd [term 2 FOLLOWER]: Refusing update from remote peer a47e5c56f13843018ac584914a8d4707: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 20:46:16.266851 10138 consensus_queue.cc:1035] T 9258667ccc3e4ef8b26fe75531dd4591 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:16.267362 11512 raft_consensus.cc:2932] T 9258667ccc3e4ef8b26fe75531dd4591 P a47e5c56f13843018ac584914a8d4707 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } } }
I20240502 20:46:16.268939 4222 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9258667ccc3e4ef8b26fe75531dd4591 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:16.270020 4374 raft_consensus.cc:2932] T 9258667ccc3e4ef8b26fe75531dd4591 P c92e6e46782849479c8a0c6d82276fb2 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } } }
I20240502 20:46:16.270486 4232 catalog_manager.cc:5484] T 9258667ccc3e4ef8b26fe75531dd4591 P a47e5c56f13843018ac584914a8d4707 reported cstate change: config changed from index 3 to 5, VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) evicted. New cstate: current_term: 2 leader_uuid: "a47e5c56f13843018ac584914a8d4707" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:16.272655 4768 raft_consensus.cc:2932] T 9258667ccc3e4ef8b26fe75531dd4591 P ed47c023452a497192f35d2f617bf9bd [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } } }
I20240502 20:46:16.273365 6681 tablet_service.cc:1514] Processing DeleteTablet for tablet 9258667ccc3e4ef8b26fe75531dd4591 with delete_type TABLET_DATA_TOMBSTONED (TS 7845c397230d49358ba2fad9db53f97c not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:37506
I20240502 20:46:16.274986 11609 tablet_replica.cc:330] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c: stopping tablet replica
I20240502 20:46:16.275189 11609 raft_consensus.cc:2229] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:16.275352 11609 raft_consensus.cc:2258] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:16.276131 11609 ts_tablet_manager.cc:1875] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:16.278143 11609 ts_tablet_manager.cc:1888] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240502 20:46:16.278216 11609 log.cc:1196] T 9258667ccc3e4ef8b26fe75531dd4591 P 7845c397230d49358ba2fad9db53f97c: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/wal/wals/9258667ccc3e4ef8b26fe75531dd4591
I20240502 20:46:16.278736 4221 catalog_manager.cc:4830] TS 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): tablet 9258667ccc3e4ef8b26fe75531dd4591 (table rebalancer_extra_table_14 [id=319cff57d33e47f69faa288db2be515b]) successfully deleted
I20240502 20:46:16.300086 4228 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 20:46:16.300251 4228 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 20:46:16.302381 4228 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: Sending DeleteTablet for 3 replicas of tablet 7037d9125d724460aac2482ae1e3dbde
I20240502 20:46:16.303063 6947 tablet_service.cc:1514] Processing DeleteTablet for tablet 7037d9125d724460aac2482ae1e3dbde with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:16 UTC) from {username='slave'} at 127.0.0.1:45156
I20240502 20:46:16.303063 7079 tablet_service.cc:1514] Processing DeleteTablet for tablet 7037d9125d724460aac2482ae1e3dbde with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:16 UTC) from {username='slave'} at 127.0.0.1:59162
I20240502 20:46:16.303164 6681 tablet_service.cc:1514] Processing DeleteTablet for tablet 7037d9125d724460aac2482ae1e3dbde with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:16 UTC) from {username='slave'} at 127.0.0.1:37506
I20240502 20:46:16.303413 11609 tablet_replica.cc:330] T 7037d9125d724460aac2482ae1e3dbde P 7845c397230d49358ba2fad9db53f97c: stopping tablet replica
I20240502 20:46:16.303432 11611 tablet_replica.cc:330] T 7037d9125d724460aac2482ae1e3dbde P a47e5c56f13843018ac584914a8d4707: stopping tablet replica
I20240502 20:46:16.303555 11610 tablet_replica.cc:330] T 7037d9125d724460aac2482ae1e3dbde P 201d2b6485084e7dad2cbb722af94779: stopping tablet replica
I20240502 20:46:16.303668 11611 raft_consensus.cc:2229] T 7037d9125d724460aac2482ae1e3dbde P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:16.303673 11609 raft_consensus.cc:2229] T 7037d9125d724460aac2482ae1e3dbde P 7845c397230d49358ba2fad9db53f97c [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:16.303732 11610 raft_consensus.cc:2229] T 7037d9125d724460aac2482ae1e3dbde P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:16.303848 11611 raft_consensus.cc:2258] T 7037d9125d724460aac2482ae1e3dbde P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:16.303884 11610 raft_consensus.cc:2258] T 7037d9125d724460aac2482ae1e3dbde P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:16.304412 11609 raft_consensus.cc:2258] T 7037d9125d724460aac2482ae1e3dbde P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:16.304495 11610 ts_tablet_manager.cc:1875] T 7037d9125d724460aac2482ae1e3dbde P 201d2b6485084e7dad2cbb722af94779: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:16.304935 11609 ts_tablet_manager.cc:1875] T 7037d9125d724460aac2482ae1e3dbde P 7845c397230d49358ba2fad9db53f97c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:16.305402 11611 ts_tablet_manager.cc:1875] T 7037d9125d724460aac2482ae1e3dbde P a47e5c56f13843018ac584914a8d4707: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:16.306651 11610 ts_tablet_manager.cc:1888] T 7037d9125d724460aac2482ae1e3dbde P 201d2b6485084e7dad2cbb722af94779: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:16.306778 11610 log.cc:1196] T 7037d9125d724460aac2482ae1e3dbde P 201d2b6485084e7dad2cbb722af94779: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-7/wal/wals/7037d9125d724460aac2482ae1e3dbde
I20240502 20:46:16.307521 11610 ts_tablet_manager.cc:1909] T 7037d9125d724460aac2482ae1e3dbde P 201d2b6485084e7dad2cbb722af94779: Deleting consensus metadata
I20240502 20:46:16.307754 11611 ts_tablet_manager.cc:1888] T 7037d9125d724460aac2482ae1e3dbde P a47e5c56f13843018ac584914a8d4707: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:16.307845 11611 log.cc:1196] T 7037d9125d724460aac2482ae1e3dbde P a47e5c56f13843018ac584914a8d4707: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/wal/wals/7037d9125d724460aac2482ae1e3dbde
I20240502 20:46:16.307947 11609 ts_tablet_manager.cc:1888] T 7037d9125d724460aac2482ae1e3dbde P 7845c397230d49358ba2fad9db53f97c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:16.308028 11609 log.cc:1196] T 7037d9125d724460aac2482ae1e3dbde P 7845c397230d49358ba2fad9db53f97c: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-4/wal/wals/7037d9125d724460aac2482ae1e3dbde
I20240502 20:46:16.308168 11611 ts_tablet_manager.cc:1909] T 7037d9125d724460aac2482ae1e3dbde P a47e5c56f13843018ac584914a8d4707: Deleting consensus metadata
I20240502 20:46:16.308211 4219 catalog_manager.cc:4830] TS 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): tablet 7037d9125d724460aac2482ae1e3dbde (table rebalancer_dynamic_table_new_name [id=dd726af14b9f4e538960924b7e78da85]) successfully deleted
I20240502 20:46:16.308297 11609 ts_tablet_manager.cc:1909] T 7037d9125d724460aac2482ae1e3dbde P 7845c397230d49358ba2fad9db53f97c: Deleting consensus metadata
I20240502 20:46:16.308866 4222 catalog_manager.cc:4830] TS a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): tablet 7037d9125d724460aac2482ae1e3dbde (table rebalancer_dynamic_table_new_name [id=dd726af14b9f4e538960924b7e78da85]) successfully deleted
I20240502 20:46:16.308866 4221 catalog_manager.cc:4830] TS 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): tablet 7037d9125d724460aac2482ae1e3dbde (table rebalancer_dynamic_table_new_name [id=dd726af14b9f4e538960924b7e78da85]) successfully deleted
I20240502 20:46:16.370112 11070 raft_consensus.cc:482] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:16.370229 11070 raft_consensus.cc:504] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:16.370676 11070 leader_election.cc:288] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867)
I20240502 20:46:16.371073 6702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cedd1c79256a4f32a73b6a88244e6181" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7845c397230d49358ba2fad9db53f97c" is_pre_election: true
I20240502 20:46:16.371219 6702 raft_consensus.cc:2454] T cedd1c79256a4f32a73b6a88244e6181 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 0.
I20240502 20:46:16.371534 6835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cedd1c79256a4f32a73b6a88244e6181" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833" is_pre_election: true
I20240502 20:46:16.371673 6835 raft_consensus.cc:2454] T cedd1c79256a4f32a73b6a88244e6181 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 0.
I20240502 20:46:16.371939 4312 leader_election.cc:302] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7845c397230d49358ba2fad9db53f97c, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:46:16.372085 11070 raft_consensus.cc:2781] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:16.372130 11070 raft_consensus.cc:482] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:16.372184 11070 raft_consensus.cc:3037] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:16.373476 11070 raft_consensus.cc:504] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:16.373843 11070 leader_election.cc:288] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 election: Requested vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867)
I20240502 20:46:16.374141 6835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cedd1c79256a4f32a73b6a88244e6181" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833"
I20240502 20:46:16.374235 6702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cedd1c79256a4f32a73b6a88244e6181" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7845c397230d49358ba2fad9db53f97c"
I20240502 20:46:16.374303 6835 raft_consensus.cc:3037] T cedd1c79256a4f32a73b6a88244e6181 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:16.374352 6702 raft_consensus.cc:3037] T cedd1c79256a4f32a73b6a88244e6181 P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:16.375303 6835 raft_consensus.cc:2454] T cedd1c79256a4f32a73b6a88244e6181 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 1.
I20240502 20:46:16.375319 6702 raft_consensus.cc:2454] T cedd1c79256a4f32a73b6a88244e6181 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 1.
I20240502 20:46:16.375591 4312 leader_election.cc:302] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7845c397230d49358ba2fad9db53f97c, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:46:16.375820 11070 raft_consensus.cc:2781] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:16.375909 11070 raft_consensus.cc:686] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEADER]: Becoming Leader. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Running, Role: LEADER
I20240502 20:46:16.376045 11070 consensus_queue.cc:237] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:16.378126 4228 catalog_manager.cc:5484] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65). New cstate: current_term: 1 leader_uuid: "c92e6e46782849479c8a0c6d82276fb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:16.403939 4232 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41812:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 20:46:16.408836 4750 tablet_service.cc:1467] Processing CreateTablet for tablet e74e4524e9a94450926d7d48e5d5d014 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6f1c4de0b1d94a7ea89e8d9fc0e39601]), partition=
I20240502 20:46:16.409036 6815 tablet_service.cc:1467] Processing CreateTablet for tablet e74e4524e9a94450926d7d48e5d5d014 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6f1c4de0b1d94a7ea89e8d9fc0e39601]), partition=
I20240502 20:46:16.409052 6947 tablet_service.cc:1467] Processing CreateTablet for tablet e74e4524e9a94450926d7d48e5d5d014 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6f1c4de0b1d94a7ea89e8d9fc0e39601]), partition=
I20240502 20:46:16.409216 4750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e74e4524e9a94450926d7d48e5d5d014. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:16.409284 6815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e74e4524e9a94450926d7d48e5d5d014. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:16.409288 6947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e74e4524e9a94450926d7d48e5d5d014. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:16.411106 11551 tablet_bootstrap.cc:492] T e74e4524e9a94450926d7d48e5d5d014 P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap starting.
I20240502 20:46:16.411368 11557 tablet_bootstrap.cc:492] T e74e4524e9a94450926d7d48e5d5d014 P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:46:16.411883 11643 tablet_bootstrap.cc:492] T e74e4524e9a94450926d7d48e5d5d014 P a47e5c56f13843018ac584914a8d4707: Bootstrap starting.
I20240502 20:46:16.412030 11551 tablet_bootstrap.cc:654] T e74e4524e9a94450926d7d48e5d5d014 P a4bf1df7fa6b4948be502c182bfbb833: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:16.412237 11557 tablet_bootstrap.cc:654] T e74e4524e9a94450926d7d48e5d5d014 P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:16.412822 11643 tablet_bootstrap.cc:654] T e74e4524e9a94450926d7d48e5d5d014 P a47e5c56f13843018ac584914a8d4707: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:16.413458 11557 tablet_bootstrap.cc:492] T e74e4524e9a94450926d7d48e5d5d014 P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:46:16.413556 11557 ts_tablet_manager.cc:1367] T e74e4524e9a94450926d7d48e5d5d014 P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:16.413894 11551 tablet_bootstrap.cc:492] T e74e4524e9a94450926d7d48e5d5d014 P a4bf1df7fa6b4948be502c182bfbb833: No bootstrap required, opened a new log
I20240502 20:46:16.413952 11551 ts_tablet_manager.cc:1367] T e74e4524e9a94450926d7d48e5d5d014 P a4bf1df7fa6b4948be502c182bfbb833: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 20:46:16.413952 11557 raft_consensus.cc:348] T e74e4524e9a94450926d7d48e5d5d014 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:16.414067 11557 raft_consensus.cc:374] T e74e4524e9a94450926d7d48e5d5d014 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:16.414100 11557 raft_consensus.cc:729] T e74e4524e9a94450926d7d48e5d5d014 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:46:16.414223 11557 consensus_queue.cc:260] T e74e4524e9a94450926d7d48e5d5d014 P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:16.414345 11551 raft_consensus.cc:348] T e74e4524e9a94450926d7d48e5d5d014 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:16.414448 11551 raft_consensus.cc:374] T e74e4524e9a94450926d7d48e5d5d014 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:16.414482 11551 raft_consensus.cc:729] T e74e4524e9a94450926d7d48e5d5d014 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Initialized, Role: FOLLOWER
I20240502 20:46:16.414494 11557 ts_tablet_manager.cc:1398] T e74e4524e9a94450926d7d48e5d5d014 P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:16.414599 11551 consensus_queue.cc:260] T e74e4524e9a94450926d7d48e5d5d014 P a4bf1df7fa6b4948be502c182bfbb833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:16.414827 11551 ts_tablet_manager.cc:1398] T e74e4524e9a94450926d7d48e5d5d014 P a4bf1df7fa6b4948be502c182bfbb833: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:16.417057 11643 tablet_bootstrap.cc:492] T e74e4524e9a94450926d7d48e5d5d014 P a47e5c56f13843018ac584914a8d4707: No bootstrap required, opened a new log
I20240502 20:46:16.417148 11643 ts_tablet_manager.cc:1367] T e74e4524e9a94450926d7d48e5d5d014 P a47e5c56f13843018ac584914a8d4707: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240502 20:46:16.417601 11643 raft_consensus.cc:348] T e74e4524e9a94450926d7d48e5d5d014 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:16.417750 11643 raft_consensus.cc:374] T e74e4524e9a94450926d7d48e5d5d014 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:16.417827 11643 raft_consensus.cc:729] T e74e4524e9a94450926d7d48e5d5d014 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Initialized, Role: FOLLOWER
I20240502 20:46:16.417968 11643 consensus_queue.cc:260] T e74e4524e9a94450926d7d48e5d5d014 P a47e5c56f13843018ac584914a8d4707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:16.418414 11643 ts_tablet_manager.cc:1398] T e74e4524e9a94450926d7d48e5d5d014 P a47e5c56f13843018ac584914a8d4707: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:16.478523 10504 consensus_queue.cc:1035] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:16.482093 10922 consensus_queue.cc:1035] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:16.504905 11428 consensus_queue.cc:1035] T abe3bf6cb45a438fbd10bd5e78febc32 P 201d2b6485084e7dad2cbb722af94779 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:16.508051 11428 consensus_queue.cc:1035] T abe3bf6cb45a438fbd10bd5e78febc32 P 201d2b6485084e7dad2cbb722af94779 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:16.556769 11081 raft_consensus.cc:482] T e74e4524e9a94450926d7d48e5d5d014 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:16.556907 11081 raft_consensus.cc:504] T e74e4524e9a94450926d7d48e5d5d014 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:16.557348 11081 leader_election.cc:288] T e74e4524e9a94450926d7d48e5d5d014 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
I20240502 20:46:16.557767 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e74e4524e9a94450926d7d48e5d5d014" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd" is_pre_election: true
I20240502 20:46:16.557814 6957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e74e4524e9a94450926d7d48e5d5d014" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47e5c56f13843018ac584914a8d4707" is_pre_election: true
I20240502 20:46:16.557912 4768 raft_consensus.cc:2454] T e74e4524e9a94450926d7d48e5d5d014 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 0.
I20240502 20:46:16.557966 6957 raft_consensus.cc:2454] T e74e4524e9a94450926d7d48e5d5d014 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 0.
I20240502 20:46:16.558228 6771 leader_election.cc:302] T e74e4524e9a94450926d7d48e5d5d014 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4bf1df7fa6b4948be502c182bfbb833, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:46:16.558394 11081 raft_consensus.cc:2781] T e74e4524e9a94450926d7d48e5d5d014 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:16.558450 11081 raft_consensus.cc:482] T e74e4524e9a94450926d7d48e5d5d014 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:16.558504 11081 raft_consensus.cc:3037] T e74e4524e9a94450926d7d48e5d5d014 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:16.559336 11081 raft_consensus.cc:504] T e74e4524e9a94450926d7d48e5d5d014 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:16.559648 11081 leader_election.cc:288] T e74e4524e9a94450926d7d48e5d5d014 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 election: Requested vote from peers ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
I20240502 20:46:16.559983 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e74e4524e9a94450926d7d48e5d5d014" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd"
I20240502 20:46:16.560009 6957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e74e4524e9a94450926d7d48e5d5d014" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47e5c56f13843018ac584914a8d4707"
I20240502 20:46:16.560108 6957 raft_consensus.cc:3037] T e74e4524e9a94450926d7d48e5d5d014 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:16.560109 4768 raft_consensus.cc:3037] T e74e4524e9a94450926d7d48e5d5d014 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:16.561117 6957 raft_consensus.cc:2454] T e74e4524e9a94450926d7d48e5d5d014 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 1.
I20240502 20:46:16.561117 4768 raft_consensus.cc:2454] T e74e4524e9a94450926d7d48e5d5d014 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 1.
I20240502 20:46:16.561407 6771 leader_election.cc:302] T e74e4524e9a94450926d7d48e5d5d014 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a47e5c56f13843018ac584914a8d4707, a4bf1df7fa6b4948be502c182bfbb833; no voters:
I20240502 20:46:16.561563 11081 raft_consensus.cc:2781] T e74e4524e9a94450926d7d48e5d5d014 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:16.561655 11081 raft_consensus.cc:686] T e74e4524e9a94450926d7d48e5d5d014 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEADER]: Becoming Leader. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Running, Role: LEADER
I20240502 20:46:16.561820 11081 consensus_queue.cc:237] T e74e4524e9a94450926d7d48e5d5d014 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:16.564018 4228 catalog_manager.cc:5484] T e74e4524e9a94450926d7d48e5d5d014 P a4bf1df7fa6b4948be502c182bfbb833 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70). New cstate: current_term: 1 leader_uuid: "a4bf1df7fa6b4948be502c182bfbb833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: HEALTHY } } }
I20240502 20:46:16.655862 6835 raft_consensus.cc:1261] T 28da7347f982468281570ffa7503a854 P a4bf1df7fa6b4948be502c182bfbb833 [term 2 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 20:46:16.656277 11600 consensus_queue.cc:1035] T 28da7347f982468281570ffa7503a854 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:16.702257 4228 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 20:46:16.704449 6957 raft_consensus.cc:1261] T 28da7347f982468281570ffa7503a854 P a47e5c56f13843018ac584914a8d4707 [term 2 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 20:46:16.704947 11600 consensus_queue.cc:1035] T 28da7347f982468281570ffa7503a854 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:16.706863 6957 raft_consensus.cc:1261] T e74e4524e9a94450926d7d48e5d5d014 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Refusing update from remote peer a4bf1df7fa6b4948be502c182bfbb833: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:16.706983 4768 raft_consensus.cc:1261] T e74e4524e9a94450926d7d48e5d5d014 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer a4bf1df7fa6b4948be502c182bfbb833: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:16.707631 11605 consensus_queue.cc:1035] T e74e4524e9a94450926d7d48e5d5d014 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:16.707890 4638 raft_consensus.cc:1261] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 2 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 20:46:16.709815 10873 consensus_queue.cc:1035] T 28da7347f982468281570ffa7503a854 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:16.710810 11605 consensus_queue.cc:1035] T e74e4524e9a94450926d7d48e5d5d014 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:16.710950 4507 consensus_queue.cc:237] T 28da7347f982468281570ffa7503a854 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } }
I20240502 20:46:16.712685 11664 tablet.cc:1722] T e74e4524e9a94450926d7d48e5d5d014 P a4bf1df7fa6b4948be502c182bfbb833: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 20:46:16.713092 11664 tablet.cc:1620] T e74e4524e9a94450926d7d48e5d5d014 P a4bf1df7fa6b4948be502c182bfbb833: MemRowSet was empty: no flush needed.
I20240502 20:46:16.713989 6957 raft_consensus.cc:1261] T 28da7347f982468281570ffa7503a854 P a47e5c56f13843018ac584914a8d4707 [term 2 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 20:46:16.714594 10873 consensus_queue.cc:1035] T 28da7347f982468281570ffa7503a854 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:46:16.714730 4232 catalog_manager.cc:5576] TS ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193) does not have the latest schema for tablet e74e4524e9a94450926d7d48e5d5d014 (table rebalancer_dynamic_table [id=6f1c4de0b1d94a7ea89e8d9fc0e39601]). Expected version 1 got 0
I20240502 20:46:16.715332 4228 catalog_manager.cc:5576] TS a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841) does not have the latest schema for tablet e74e4524e9a94450926d7d48e5d5d014 (table rebalancer_dynamic_table [id=6f1c4de0b1d94a7ea89e8d9fc0e39601]). Expected version 1 got 0
I20240502 20:46:16.716200 11600 raft_consensus.cc:2932] T 28da7347f982468281570ffa7503a854 P c3d26eafd51547acb780d4aac5ebe7cc [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:16.718214 4228 catalog_manager.cc:5576] TS a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841) does not have the latest schema for tablet e74e4524e9a94450926d7d48e5d5d014 (table rebalancer_dynamic_table [id=6f1c4de0b1d94a7ea89e8d9fc0e39601]). Expected version 1 got 0
I20240502 20:46:16.719023 4221 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 28da7347f982468281570ffa7503a854 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:16.718974 6967 raft_consensus.cc:2932] T 28da7347f982468281570ffa7503a854 P a47e5c56f13843018ac584914a8d4707 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:16.719417 4232 catalog_manager.cc:5484] T 28da7347f982468281570ffa7503a854 P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: config changed from index 8 to 10, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New cstate: current_term: 2 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:16.721709 6835 raft_consensus.cc:1261] T 28da7347f982468281570ffa7503a854 P a4bf1df7fa6b4948be502c182bfbb833 [term 2 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 20:46:16.724539 4618 tablet_service.cc:1514] Processing DeleteTablet for tablet 28da7347f982468281570ffa7503a854 with delete_type TABLET_DATA_TOMBSTONED (TS 923cfab926e7443f8ffd7b21ad0a9d91 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:57068
I20240502 20:46:16.724637 4227 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: rebalancer_dynamic_table [id=6f1c4de0b1d94a7ea89e8d9fc0e39601] alter complete (version 1)
I20240502 20:46:16.724783 11606 tablet_replica.cc:330] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91: stopping tablet replica
I20240502 20:46:16.724938 11606 raft_consensus.cc:2229] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:16.725092 11606 raft_consensus.cc:2258] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:16.725347 10873 consensus_queue.cc:1035] T 28da7347f982468281570ffa7503a854 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:46:16.725528 4228 catalog_manager.cc:5576] TS ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193) does not have the latest schema for tablet e74e4524e9a94450926d7d48e5d5d014 (table rebalancer_dynamic_table [id=6f1c4de0b1d94a7ea89e8d9fc0e39601]). Expected version 1 got 0
I20240502 20:46:16.726063 11666 tablet.cc:1722] T e74e4524e9a94450926d7d48e5d5d014 P a47e5c56f13843018ac584914a8d4707: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 20:46:16.726361 11606 ts_tablet_manager.cc:1875] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:16.726387 11667 tablet.cc:1722] T e74e4524e9a94450926d7d48e5d5d014 P ed47c023452a497192f35d2f617bf9bd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 20:46:16.726496 6835 raft_consensus.cc:2932] T 28da7347f982468281570ffa7503a854 P a4bf1df7fa6b4948be502c182bfbb833 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { promote: false } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } }
I20240502 20:46:16.726722 11667 tablet.cc:1620] T e74e4524e9a94450926d7d48e5d5d014 P ed47c023452a497192f35d2f617bf9bd: MemRowSet was empty: no flush needed.
I20240502 20:46:16.727151 11666 tablet.cc:1620] T e74e4524e9a94450926d7d48e5d5d014 P a47e5c56f13843018ac584914a8d4707: MemRowSet was empty: no flush needed.
I20240502 20:46:16.728565 11606 ts_tablet_manager.cc:1888] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 20:46:16.728637 11606 log.cc:1196] T 28da7347f982468281570ffa7503a854 P 923cfab926e7443f8ffd7b21ad0a9d91: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-2/wal/wals/28da7347f982468281570ffa7503a854
I20240502 20:46:16.729163 4219 catalog_manager.cc:4830] TS 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141): tablet 28da7347f982468281570ffa7503a854 (table rebalance_test_table_1 [id=3a26d60c6de746e891d970c2ce9c55cc]) successfully deleted
I20240502 20:46:16.730796 4618 tablet_service.cc:1514] Processing DeleteTablet for tablet 28da7347f982468281570ffa7503a854 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 8 (current committed config index is 10)) from {username='slave'} at 127.0.0.1:57068
I20240502 20:46:16.733111 4219 catalog_manager.cc:4830] TS 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141): tablet 28da7347f982468281570ffa7503a854 (table rebalance_test_table_1 [id=3a26d60c6de746e891d970c2ce9c55cc]) successfully deleted
I20240502 20:46:16.813010 11070 consensus_queue.cc:1035] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:16.816790 11608 consensus_queue.cc:1035] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:16.832501 4228 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 20:46:16.839478 11664 tablet.cc:1722] T e74e4524e9a94450926d7d48e5d5d014 P a4bf1df7fa6b4948be502c182bfbb833: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 20:46:16.839816 11664 tablet.cc:1620] T e74e4524e9a94450926d7d48e5d5d014 P a4bf1df7fa6b4948be502c182bfbb833: MemRowSet was empty: no flush needed.
I20240502 20:46:16.841228 11667 tablet.cc:1722] T e74e4524e9a94450926d7d48e5d5d014 P ed47c023452a497192f35d2f617bf9bd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 20:46:16.841528 11667 tablet.cc:1620] T e74e4524e9a94450926d7d48e5d5d014 P ed47c023452a497192f35d2f617bf9bd: MemRowSet was empty: no flush needed.
I20240502 20:46:16.842114 11666 tablet.cc:1722] T e74e4524e9a94450926d7d48e5d5d014 P a47e5c56f13843018ac584914a8d4707: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 20:46:16.842386 11666 tablet.cc:1620] T e74e4524e9a94450926d7d48e5d5d014 P a47e5c56f13843018ac584914a8d4707: MemRowSet was empty: no flush needed.
I20240502 20:46:16.846071 4228 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: rebalancer_dynamic_table [id=6f1c4de0b1d94a7ea89e8d9fc0e39601] alter complete (version 2)
I20240502 20:46:16.915869 4235 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41812:
name: "rebalancer_extra_table_26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:46:16.925472 4618 tablet_service.cc:1467] Processing CreateTablet for tablet 172976cb61a649d3995425d9859b432c (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=752e5ef7eb0c49a3a048153432926088]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:16.925607 6947 tablet_service.cc:1467] Processing CreateTablet for tablet 5ed65a0bf8c0469eb6ea91bcf4f38daa (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=752e5ef7eb0c49a3a048153432926088]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:16.925814 4618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 172976cb61a649d3995425d9859b432c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:16.925892 6947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ed65a0bf8c0469eb6ea91bcf4f38daa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:16.927198 4354 tablet_service.cc:1467] Processing CreateTablet for tablet 5ed65a0bf8c0469eb6ea91bcf4f38daa (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=752e5ef7eb0c49a3a048153432926088]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:16.927484 4354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ed65a0bf8c0469eb6ea91bcf4f38daa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:16.928014 11731 tablet_bootstrap.cc:492] T 172976cb61a649d3995425d9859b432c P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:46:16.929240 11732 tablet_bootstrap.cc:492] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P a47e5c56f13843018ac584914a8d4707: Bootstrap starting.
I20240502 20:46:16.930085 11731 tablet_bootstrap.cc:654] T 172976cb61a649d3995425d9859b432c P 923cfab926e7443f8ffd7b21ad0a9d91: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:16.930191 4356 tablet_service.cc:1467] Processing CreateTablet for tablet 172976cb61a649d3995425d9859b432c (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=752e5ef7eb0c49a3a048153432926088]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:16.930310 11732 tablet_bootstrap.cc:654] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P a47e5c56f13843018ac584914a8d4707: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:16.930434 4356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 172976cb61a649d3995425d9859b432c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:16.931787 11732 tablet_bootstrap.cc:492] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P a47e5c56f13843018ac584914a8d4707: No bootstrap required, opened a new log
I20240502 20:46:16.931869 11732 ts_tablet_manager.cc:1367] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P a47e5c56f13843018ac584914a8d4707: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:46:16.932143 11731 tablet_bootstrap.cc:492] T 172976cb61a649d3995425d9859b432c P 923cfab926e7443f8ffd7b21ad0a9d91: No bootstrap required, opened a new log
I20240502 20:46:16.932217 11731 ts_tablet_manager.cc:1367] T 172976cb61a649d3995425d9859b432c P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 20:46:16.932343 11732 raft_consensus.cc:348] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:16.932440 11732 raft_consensus.cc:374] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:16.932472 11732 raft_consensus.cc:729] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Initialized, Role: FOLLOWER
I20240502 20:46:16.932510 7079 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:46:16.932610 11732 consensus_queue.cc:260] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P a47e5c56f13843018ac584914a8d4707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:16.932722 11731 raft_consensus.cc:348] T 172976cb61a649d3995425d9859b432c P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:16.932808 11731 raft_consensus.cc:374] T 172976cb61a649d3995425d9859b432c P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:16.932842 11731 raft_consensus.cc:729] T 172976cb61a649d3995425d9859b432c P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: FOLLOWER
I20240502 20:46:16.932919 11732 ts_tablet_manager.cc:1398] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P a47e5c56f13843018ac584914a8d4707: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:16.932958 11731 consensus_queue.cc:260] T 172976cb61a649d3995425d9859b432c P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:16.933243 11731 ts_tablet_manager.cc:1398] T 172976cb61a649d3995425d9859b432c P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:16.936652 6815 tablet_service.cc:1467] Processing CreateTablet for tablet dff789a7a50c4ff0a1eb29489ac7901a (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=752e5ef7eb0c49a3a048153432926088]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:16.936949 6815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dff789a7a50c4ff0a1eb29489ac7901a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:16.938158 6681 tablet_service.cc:1467] Processing CreateTablet for tablet 5ed65a0bf8c0469eb6ea91bcf4f38daa (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=752e5ef7eb0c49a3a048153432926088]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:16.938416 6681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ed65a0bf8c0469eb6ea91bcf4f38daa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:16.938743 6683 tablet_service.cc:1467] Processing CreateTablet for tablet dff789a7a50c4ff0a1eb29489ac7901a (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=752e5ef7eb0c49a3a048153432926088]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:16.938989 6683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dff789a7a50c4ff0a1eb29489ac7901a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:16.940567 11733 tablet_bootstrap.cc:492] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:46:16.941924 11733 tablet_bootstrap.cc:654] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:16.943086 11733 tablet_bootstrap.cc:492] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:46:16.943185 11733 ts_tablet_manager.cc:1367] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 20:46:16.943326 11737 tablet_bootstrap.cc:492] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P 7845c397230d49358ba2fad9db53f97c: Bootstrap starting.
I20240502 20:46:16.943336 4487 tablet_service.cc:1467] Processing CreateTablet for tablet dff789a7a50c4ff0a1eb29489ac7901a (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=752e5ef7eb0c49a3a048153432926088]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:16.943578 4487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dff789a7a50c4ff0a1eb29489ac7901a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:16.943603 11733 raft_consensus.cc:348] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:16.943701 11733 raft_consensus.cc:374] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:16.943729 11733 raft_consensus.cc:729] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:46:16.943841 11733 consensus_queue.cc:260] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:16.944115 11733 ts_tablet_manager.cc:1398] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:16.944303 11733 tablet_bootstrap.cc:492] T 172976cb61a649d3995425d9859b432c P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:46:16.945127 11737 tablet_bootstrap.cc:654] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P 7845c397230d49358ba2fad9db53f97c: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:16.946362 11733 tablet_bootstrap.cc:654] T 172976cb61a649d3995425d9859b432c P c92e6e46782849479c8a0c6d82276fb2: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:16.947656 11733 tablet_bootstrap.cc:492] T 172976cb61a649d3995425d9859b432c P c92e6e46782849479c8a0c6d82276fb2: No bootstrap required, opened a new log
I20240502 20:46:16.947665 11737 tablet_bootstrap.cc:492] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P 7845c397230d49358ba2fad9db53f97c: No bootstrap required, opened a new log
I20240502 20:46:16.947739 11737 ts_tablet_manager.cc:1367] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P 7845c397230d49358ba2fad9db53f97c: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240502 20:46:16.947744 11733 ts_tablet_manager.cc:1367] T 172976cb61a649d3995425d9859b432c P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 20:46:16.948099 11733 raft_consensus.cc:348] T 172976cb61a649d3995425d9859b432c P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:16.948184 11733 raft_consensus.cc:374] T 172976cb61a649d3995425d9859b432c P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:16.948204 11733 raft_consensus.cc:729] T 172976cb61a649d3995425d9859b432c P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: FOLLOWER
I20240502 20:46:16.948309 11733 consensus_queue.cc:260] T 172976cb61a649d3995425d9859b432c P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:16.948333 11737 raft_consensus.cc:348] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:16.948426 11737 raft_consensus.cc:374] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:16.948451 11737 raft_consensus.cc:729] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Initialized, Role: FOLLOWER
I20240502 20:46:16.948573 11733 ts_tablet_manager.cc:1398] T 172976cb61a649d3995425d9859b432c P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:16.948565 11737 consensus_queue.cc:260] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P 7845c397230d49358ba2fad9db53f97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:16.948843 11737 ts_tablet_manager.cc:1398] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P 7845c397230d49358ba2fad9db53f97c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 20:46:16.949009 11737 tablet_bootstrap.cc:492] T dff789a7a50c4ff0a1eb29489ac7901a P 7845c397230d49358ba2fad9db53f97c: Bootstrap starting.
I20240502 20:46:16.949904 11737 tablet_bootstrap.cc:654] T dff789a7a50c4ff0a1eb29489ac7901a P 7845c397230d49358ba2fad9db53f97c: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:16.936702 6802 tablet_service.cc:1467] Processing CreateTablet for tablet 172976cb61a649d3995425d9859b432c (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=752e5ef7eb0c49a3a048153432926088]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:16.950543 4230 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 20:46:16.950765 6802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 172976cb61a649d3995425d9859b432c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:16.951377 11737 tablet_bootstrap.cc:492] T dff789a7a50c4ff0a1eb29489ac7901a P 7845c397230d49358ba2fad9db53f97c: No bootstrap required, opened a new log
I20240502 20:46:16.951447 11737 ts_tablet_manager.cc:1367] T dff789a7a50c4ff0a1eb29489ac7901a P 7845c397230d49358ba2fad9db53f97c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:16.951913 11737 raft_consensus.cc:348] T dff789a7a50c4ff0a1eb29489ac7901a P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:16.952009 11737 raft_consensus.cc:374] T dff789a7a50c4ff0a1eb29489ac7901a P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:16.952044 11737 raft_consensus.cc:729] T dff789a7a50c4ff0a1eb29489ac7901a P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Initialized, Role: FOLLOWER
I20240502 20:46:16.952181 11737 consensus_queue.cc:260] T dff789a7a50c4ff0a1eb29489ac7901a P 7845c397230d49358ba2fad9db53f97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:16.952436 11737 ts_tablet_manager.cc:1398] T dff789a7a50c4ff0a1eb29489ac7901a P 7845c397230d49358ba2fad9db53f97c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:16.953822 4356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:46:16.957470 6815 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:16.958911 4618 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 20:46:16.964028 6683 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:16.965459 4487 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 20:46:16.968218 11664 tablet.cc:1722] T e74e4524e9a94450926d7d48e5d5d014 P a4bf1df7fa6b4948be502c182bfbb833: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 20:46:16.971452 11667 tablet.cc:1722] T e74e4524e9a94450926d7d48e5d5d014 P ed47c023452a497192f35d2f617bf9bd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 20:46:16.975127 11736 tablet_bootstrap.cc:492] T dff789a7a50c4ff0a1eb29489ac7901a P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap starting.
I20240502 20:46:16.975692 11739 tablet_bootstrap.cc:492] T dff789a7a50c4ff0a1eb29489ac7901a P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:46:16.976378 11666 tablet.cc:1722] T e74e4524e9a94450926d7d48e5d5d014 P a47e5c56f13843018ac584914a8d4707: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 20:46:16.977274 11739 tablet_bootstrap.cc:654] T dff789a7a50c4ff0a1eb29489ac7901a P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:16.978417 11736 tablet_bootstrap.cc:654] T dff789a7a50c4ff0a1eb29489ac7901a P a4bf1df7fa6b4948be502c182bfbb833: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:16.980783 4235 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: rebalancer_dynamic_table_new_name [id=6f1c4de0b1d94a7ea89e8d9fc0e39601] alter complete (version 3)
I20240502 20:46:16.982645 6947 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 20:46:16.983078 4750 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:46:16.985078 10504 raft_consensus.cc:482] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:16.985172 10504 raft_consensus.cc:504] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:16.985555 10504 leader_election.cc:288] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:46:16.986061 4374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ed65a0bf8c0469eb6ea91bcf4f38daa" candidate_uuid: "7845c397230d49358ba2fad9db53f97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2" is_pre_election: true
I20240502 20:46:16.986240 4374 raft_consensus.cc:2454] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7845c397230d49358ba2fad9db53f97c in term 0.
I20240502 20:46:16.986527 6967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ed65a0bf8c0469eb6ea91bcf4f38daa" candidate_uuid: "7845c397230d49358ba2fad9db53f97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47e5c56f13843018ac584914a8d4707" is_pre_election: true
I20240502 20:46:16.986603 6639 leader_election.cc:302] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7845c397230d49358ba2fad9db53f97c, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:46:16.986661 6967 raft_consensus.cc:2454] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7845c397230d49358ba2fad9db53f97c in term 0.
I20240502 20:46:16.986912 10504 raft_consensus.cc:2781] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:16.986974 10504 raft_consensus.cc:482] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:16.987000 10504 raft_consensus.cc:3037] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:16.987819 10504 raft_consensus.cc:504] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:16.988127 10504 leader_election.cc:288] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 election: Requested vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:46:16.988528 4374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ed65a0bf8c0469eb6ea91bcf4f38daa" candidate_uuid: "7845c397230d49358ba2fad9db53f97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2"
I20240502 20:46:16.988641 4374 raft_consensus.cc:3037] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:16.989015 6967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ed65a0bf8c0469eb6ea91bcf4f38daa" candidate_uuid: "7845c397230d49358ba2fad9db53f97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47e5c56f13843018ac584914a8d4707"
I20240502 20:46:16.989114 6967 raft_consensus.cc:3037] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:16.989702 4374 raft_consensus.cc:2454] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7845c397230d49358ba2fad9db53f97c in term 1.
I20240502 20:46:16.989974 6639 leader_election.cc:302] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P 7845c397230d49358ba2fad9db53f97c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7845c397230d49358ba2fad9db53f97c, c92e6e46782849479c8a0c6d82276fb2; no voters:
I20240502 20:46:16.990121 6967 raft_consensus.cc:2454] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7845c397230d49358ba2fad9db53f97c in term 1.
I20240502 20:46:16.990136 10504 raft_consensus.cc:2781] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:16.990239 10504 raft_consensus.cc:686] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P 7845c397230d49358ba2fad9db53f97c [term 1 LEADER]: Becoming Leader. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Running, Role: LEADER
I20240502 20:46:16.990383 10504 consensus_queue.cc:237] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P 7845c397230d49358ba2fad9db53f97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:17.001045 11608 raft_consensus.cc:482] T 172976cb61a649d3995425d9859b432c P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:17.001164 11608 raft_consensus.cc:504] T 172976cb61a649d3995425d9859b432c P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:17.001528 11608 leader_election.cc:288] T 172976cb61a649d3995425d9859b432c P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141), a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867)
I20240502 20:46:17.002007 6831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "172976cb61a649d3995425d9859b432c" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833" is_pre_election: true
W20240502 20:46:17.002403 4311 leader_election.cc:341] T 172976cb61a649d3995425d9859b432c P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867): Illegal state: must be running to vote when last-logged opid is not known
I20240502 20:46:17.004021 4637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "172976cb61a649d3995425d9859b432c" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" is_pre_election: true
I20240502 20:46:17.004156 4637 raft_consensus.cc:2454] T 172976cb61a649d3995425d9859b432c P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 0.
I20240502 20:46:17.004451 4310 leader_election.cc:302] T 172976cb61a649d3995425d9859b432c P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, c92e6e46782849479c8a0c6d82276fb2; no voters: a4bf1df7fa6b4948be502c182bfbb833
I20240502 20:46:17.004688 11608 raft_consensus.cc:2781] T 172976cb61a649d3995425d9859b432c P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:17.004740 11608 raft_consensus.cc:482] T 172976cb61a649d3995425d9859b432c P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:17.004786 11608 raft_consensus.cc:3037] T 172976cb61a649d3995425d9859b432c P c92e6e46782849479c8a0c6d82276fb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:17.005952 11608 raft_consensus.cc:504] T 172976cb61a649d3995425d9859b432c P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:17.006282 11608 leader_election.cc:288] T 172976cb61a649d3995425d9859b432c P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 election: Requested vote from peers 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141), a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867)
I20240502 20:46:17.007205 6835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "172976cb61a649d3995425d9859b432c" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833"
W20240502 20:46:17.007531 4311 leader_election.cc:341] T 172976cb61a649d3995425d9859b432c P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867): Illegal state: must be running to vote when last-logged opid is not known
I20240502 20:46:17.010994 11739 tablet_bootstrap.cc:492] T dff789a7a50c4ff0a1eb29489ac7901a P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
I20240502 20:46:17.011099 11739 ts_tablet_manager.cc:1367] T dff789a7a50c4ff0a1eb29489ac7901a P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.000s
I20240502 20:46:17.011101 11736 tablet_bootstrap.cc:492] T dff789a7a50c4ff0a1eb29489ac7901a P a4bf1df7fa6b4948be502c182bfbb833: No bootstrap required, opened a new log
I20240502 20:46:17.011193 11736 ts_tablet_manager.cc:1367] T dff789a7a50c4ff0a1eb29489ac7901a P a4bf1df7fa6b4948be502c182bfbb833: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.000s
I20240502 20:46:17.011559 11739 raft_consensus.cc:348] T dff789a7a50c4ff0a1eb29489ac7901a P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:17.011670 11739 raft_consensus.cc:374] T dff789a7a50c4ff0a1eb29489ac7901a P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:17.011613 11736 raft_consensus.cc:348] T dff789a7a50c4ff0a1eb29489ac7901a P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:17.011699 11739 raft_consensus.cc:729] T dff789a7a50c4ff0a1eb29489ac7901a P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:46:17.011729 11736 raft_consensus.cc:374] T dff789a7a50c4ff0a1eb29489ac7901a P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:17.011757 11736 raft_consensus.cc:729] T dff789a7a50c4ff0a1eb29489ac7901a P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Initialized, Role: FOLLOWER
I20240502 20:46:17.011821 11739 consensus_queue.cc:260] T dff789a7a50c4ff0a1eb29489ac7901a P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:17.011869 11736 consensus_queue.cc:260] T dff789a7a50c4ff0a1eb29489ac7901a P a4bf1df7fa6b4948be502c182bfbb833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:17.012109 11739 ts_tablet_manager.cc:1398] T dff789a7a50c4ff0a1eb29489ac7901a P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:17.012132 11736 ts_tablet_manager.cc:1398] T dff789a7a50c4ff0a1eb29489ac7901a P a4bf1df7fa6b4948be502c182bfbb833: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:17.012282 11736 tablet_bootstrap.cc:492] T 172976cb61a649d3995425d9859b432c P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap starting.
I20240502 20:46:17.013254 11736 tablet_bootstrap.cc:654] T 172976cb61a649d3995425d9859b432c P a4bf1df7fa6b4948be502c182bfbb833: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:17.013916 4637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "172976cb61a649d3995425d9859b432c" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91"
I20240502 20:46:17.014050 4637 raft_consensus.cc:3037] T 172976cb61a649d3995425d9859b432c P 923cfab926e7443f8ffd7b21ad0a9d91 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:17.015046 4637 raft_consensus.cc:2454] T 172976cb61a649d3995425d9859b432c P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c92e6e46782849479c8a0c6d82276fb2 in term 1.
I20240502 20:46:17.015179 11736 tablet_bootstrap.cc:492] T 172976cb61a649d3995425d9859b432c P a4bf1df7fa6b4948be502c182bfbb833: No bootstrap required, opened a new log
I20240502 20:46:17.015267 11736 ts_tablet_manager.cc:1367] T 172976cb61a649d3995425d9859b432c P a4bf1df7fa6b4948be502c182bfbb833: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:46:17.015368 4310 leader_election.cc:302] T 172976cb61a649d3995425d9859b432c P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, c92e6e46782849479c8a0c6d82276fb2; no voters: a4bf1df7fa6b4948be502c182bfbb833
I20240502 20:46:16.997807 4231 catalog_manager.cc:5484] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P 7845c397230d49358ba2fad9db53f97c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7845c397230d49358ba2fad9db53f97c (127.30.32.69). New cstate: current_term: 1 leader_uuid: "7845c397230d49358ba2fad9db53f97c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:17.015587 11608 raft_consensus.cc:2781] T 172976cb61a649d3995425d9859b432c P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:17.015676 11608 raft_consensus.cc:686] T 172976cb61a649d3995425d9859b432c P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEADER]: Becoming Leader. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Running, Role: LEADER
I20240502 20:46:17.015728 11736 raft_consensus.cc:348] T 172976cb61a649d3995425d9859b432c P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:17.015841 11736 raft_consensus.cc:374] T 172976cb61a649d3995425d9859b432c P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:17.015805 11608 consensus_queue.cc:237] T 172976cb61a649d3995425d9859b432c P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:17.015872 11736 raft_consensus.cc:729] T 172976cb61a649d3995425d9859b432c P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Initialized, Role: FOLLOWER
I20240502 20:46:17.015959 11736 consensus_queue.cc:260] T 172976cb61a649d3995425d9859b432c P a4bf1df7fa6b4948be502c182bfbb833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:17.016230 11736 ts_tablet_manager.cc:1398] T 172976cb61a649d3995425d9859b432c P a4bf1df7fa6b4948be502c182bfbb833: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:17.019018 4235 catalog_manager.cc:5484] T 172976cb61a649d3995425d9859b432c P c92e6e46782849479c8a0c6d82276fb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65). New cstate: current_term: 1 leader_uuid: "c92e6e46782849479c8a0c6d82276fb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:17.066706 6702 consensus_queue.cc:237] T 6ff3c95ccfbf486fbc4bfc636afa829b P 7845c397230d49358ba2fad9db53f97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true } }
I20240502 20:46:17.068075 7098 raft_consensus.cc:1261] T 6ff3c95ccfbf486fbc4bfc636afa829b P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Refusing update from remote peer 7845c397230d49358ba2fad9db53f97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:17.068181 6835 raft_consensus.cc:1261] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Refusing update from remote peer 7845c397230d49358ba2fad9db53f97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 20:46:17.068365 6639 consensus_peers.cc:485] T 6ff3c95ccfbf486fbc4bfc636afa829b P 7845c397230d49358ba2fad9db53f97c -> Peer c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827): Couldn't send request to peer c92e6e46782849479c8a0c6d82276fb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6ff3c95ccfbf486fbc4bfc636afa829b. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:17.068611 11746 consensus_queue.cc:1035] T 6ff3c95ccfbf486fbc4bfc636afa829b P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:46:17.068785 10504 consensus_queue.cc:1035] T 6ff3c95ccfbf486fbc4bfc636afa829b P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:46:17.070915 10504 raft_consensus.cc:2932] T 6ff3c95ccfbf486fbc4bfc636afa829b P 7845c397230d49358ba2fad9db53f97c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true } } }
I20240502 20:46:17.071513 6835 raft_consensus.cc:2932] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true } } }
I20240502 20:46:17.074275 4235 catalog_manager.cc:5484] T 6ff3c95ccfbf486fbc4bfc636afa829b P 7845c397230d49358ba2fad9db53f97c reported cstate change: config changed from index -1 to 2, NON_VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) added. New cstate: current_term: 1 leader_uuid: "7845c397230d49358ba2fad9db53f97c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:17.082571 4231 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 20:46:17.082716 4231 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 20:46:17.082551 7098 raft_consensus.cc:2932] T 6ff3c95ccfbf486fbc4bfc636afa829b P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true } } }
I20240502 20:46:17.086022 4231 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: Sending DeleteTablet for 3 replicas of tablet e74e4524e9a94450926d7d48e5d5d014
I20240502 20:46:17.086577 4750 tablet_service.cc:1514] Processing DeleteTablet for tablet e74e4524e9a94450926d7d48e5d5d014 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:17 UTC) from {username='slave'} at 127.0.0.1:38526
I20240502 20:46:17.086602 6947 tablet_service.cc:1514] Processing DeleteTablet for tablet e74e4524e9a94450926d7d48e5d5d014 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:17 UTC) from {username='slave'} at 127.0.0.1:45156
I20240502 20:46:17.087157 6802 tablet_service.cc:1514] Processing DeleteTablet for tablet e74e4524e9a94450926d7d48e5d5d014 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:17 UTC) from {username='slave'} at 127.0.0.1:35290
I20240502 20:46:17.087540 11765 tablet_replica.cc:330] T e74e4524e9a94450926d7d48e5d5d014 P a4bf1df7fa6b4948be502c182bfbb833: stopping tablet replica
I20240502 20:46:17.087798 11764 tablet_replica.cc:330] T e74e4524e9a94450926d7d48e5d5d014 P ed47c023452a497192f35d2f617bf9bd: stopping tablet replica
I20240502 20:46:17.087989 11764 raft_consensus.cc:2229] T e74e4524e9a94450926d7d48e5d5d014 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:17.087997 11765 raft_consensus.cc:2229] T e74e4524e9a94450926d7d48e5d5d014 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:17.088125 11764 raft_consensus.cc:2258] T e74e4524e9a94450926d7d48e5d5d014 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:17.088203 11765 raft_consensus.cc:2258] T e74e4524e9a94450926d7d48e5d5d014 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:17.088717 11765 ts_tablet_manager.cc:1875] T e74e4524e9a94450926d7d48e5d5d014 P a4bf1df7fa6b4948be502c182bfbb833: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:17.088866 11764 ts_tablet_manager.cc:1875] T e74e4524e9a94450926d7d48e5d5d014 P ed47c023452a497192f35d2f617bf9bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:17.090011 11763 tablet_replica.cc:330] T e74e4524e9a94450926d7d48e5d5d014 P a47e5c56f13843018ac584914a8d4707: stopping tablet replica
I20240502 20:46:17.090767 11763 raft_consensus.cc:2229] T e74e4524e9a94450926d7d48e5d5d014 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:17.090974 11763 raft_consensus.cc:2258] T e74e4524e9a94450926d7d48e5d5d014 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:17.091298 11764 ts_tablet_manager.cc:1888] T e74e4524e9a94450926d7d48e5d5d014 P ed47c023452a497192f35d2f617bf9bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:17.091118 11765 ts_tablet_manager.cc:1888] T e74e4524e9a94450926d7d48e5d5d014 P a4bf1df7fa6b4948be502c182bfbb833: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:17.091435 11764 log.cc:1196] T e74e4524e9a94450926d7d48e5d5d014 P ed47c023452a497192f35d2f617bf9bd: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-3/wal/wals/e74e4524e9a94450926d7d48e5d5d014
I20240502 20:46:17.091500 11765 log.cc:1196] T e74e4524e9a94450926d7d48e5d5d014 P a4bf1df7fa6b4948be502c182bfbb833: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/wal/wals/e74e4524e9a94450926d7d48e5d5d014
I20240502 20:46:17.091748 11764 ts_tablet_manager.cc:1909] T e74e4524e9a94450926d7d48e5d5d014 P ed47c023452a497192f35d2f617bf9bd: Deleting consensus metadata
I20240502 20:46:17.091809 11765 ts_tablet_manager.cc:1909] T e74e4524e9a94450926d7d48e5d5d014 P a4bf1df7fa6b4948be502c182bfbb833: Deleting consensus metadata
I20240502 20:46:17.092514 4222 catalog_manager.cc:4830] TS ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193): tablet e74e4524e9a94450926d7d48e5d5d014 (table rebalancer_dynamic_table_new_name [id=6f1c4de0b1d94a7ea89e8d9fc0e39601]) successfully deleted
I20240502 20:46:17.092610 4220 catalog_manager.cc:4830] TS a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867): tablet e74e4524e9a94450926d7d48e5d5d014 (table rebalancer_dynamic_table_new_name [id=6f1c4de0b1d94a7ea89e8d9fc0e39601]) successfully deleted
I20240502 20:46:17.093464 11763 ts_tablet_manager.cc:1875] T e74e4524e9a94450926d7d48e5d5d014 P a47e5c56f13843018ac584914a8d4707: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:17.095599 11763 ts_tablet_manager.cc:1888] T e74e4524e9a94450926d7d48e5d5d014 P a47e5c56f13843018ac584914a8d4707: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:17.095685 11763 log.cc:1196] T e74e4524e9a94450926d7d48e5d5d014 P a47e5c56f13843018ac584914a8d4707: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/wal/wals/e74e4524e9a94450926d7d48e5d5d014
I20240502 20:46:17.096006 11763 ts_tablet_manager.cc:1909] T e74e4524e9a94450926d7d48e5d5d014 P a47e5c56f13843018ac584914a8d4707: Deleting consensus metadata
I20240502 20:46:17.096493 7098 consensus_queue.cc:237] T 2c67f965350a45f89511679d40f04cbb P 201d2b6485084e7dad2cbb722af94779 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } }
I20240502 20:46:17.096676 4222 catalog_manager.cc:4830] TS a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): tablet e74e4524e9a94450926d7d48e5d5d014 (table rebalancer_dynamic_table_new_name [id=6f1c4de0b1d94a7ea89e8d9fc0e39601]) successfully deleted
I20240502 20:46:17.097956 6835 raft_consensus.cc:1261] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Refusing update from remote peer 201d2b6485084e7dad2cbb722af94779: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:17.098374 11767 consensus_queue.cc:1035] T 2c67f965350a45f89511679d40f04cbb P 201d2b6485084e7dad2cbb722af94779 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 20:46:17.098553 7034 consensus_peers.cc:485] T 2c67f965350a45f89511679d40f04cbb P 201d2b6485084e7dad2cbb722af94779 -> Peer c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097): Couldn't send request to peer c3d26eafd51547acb780d4aac5ebe7cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2c67f965350a45f89511679d40f04cbb. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:17.098762 6967 raft_consensus.cc:1261] T 2c67f965350a45f89511679d40f04cbb P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Refusing update from remote peer 201d2b6485084e7dad2cbb722af94779: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:17.099866 11767 consensus_queue.cc:1035] T 2c67f965350a45f89511679d40f04cbb P 201d2b6485084e7dad2cbb722af94779 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:46:17.100608 11550 raft_consensus.cc:2932] T 2c67f965350a45f89511679d40f04cbb P 201d2b6485084e7dad2cbb722af94779 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } } }
I20240502 20:46:17.102054 6967 raft_consensus.cc:2932] T 2c67f965350a45f89511679d40f04cbb P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } } }
I20240502 20:46:17.102846 6835 raft_consensus.cc:2932] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } } }
I20240502 20:46:17.104494 4235 catalog_manager.cc:5484] T 2c67f965350a45f89511679d40f04cbb P 201d2b6485084e7dad2cbb722af94779 reported cstate change: config changed from index -1 to 2, NON_VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) added. New cstate: current_term: 1 leader_uuid: "201d2b6485084e7dad2cbb722af94779" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:17.113461 6702 consensus_queue.cc:237] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } }
I20240502 20:46:17.114853 4507 raft_consensus.cc:1261] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer 7845c397230d49358ba2fad9db53f97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 20:46:17.115247 6636 consensus_peers.cc:485] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P 7845c397230d49358ba2fad9db53f97c -> Peer 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141): Couldn't send request to peer 923cfab926e7443f8ffd7b21ad0a9d91. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 75e2e48cd7a4497f9ccaae4f712f3ef0. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:17.115289 11746 consensus_queue.cc:1035] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:46:17.115795 6835 raft_consensus.cc:1261] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Refusing update from remote peer 7845c397230d49358ba2fad9db53f97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:17.116200 11746 consensus_queue.cc:1035] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:46:17.117141 10922 raft_consensus.cc:2932] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P 7845c397230d49358ba2fad9db53f97c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } } }
I20240502 20:46:17.118137 4507 raft_consensus.cc:2932] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } } }
I20240502 20:46:17.118183 6835 raft_consensus.cc:2932] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } } }
I20240502 20:46:17.120795 4230 catalog_manager.cc:5484] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P 7845c397230d49358ba2fad9db53f97c reported cstate change: config changed from index -1 to 2, NON_VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) added. New cstate: current_term: 1 leader_uuid: "7845c397230d49358ba2fad9db53f97c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:17.129058 7099 consensus_queue.cc:237] T d78c02773e454ea0b79cf1282eda125b P 201d2b6485084e7dad2cbb722af94779 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } }
I20240502 20:46:17.130772 4374 raft_consensus.cc:1261] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer 201d2b6485084e7dad2cbb722af94779: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 20:46:17.131253 7034 consensus_peers.cc:485] T d78c02773e454ea0b79cf1282eda125b P 201d2b6485084e7dad2cbb722af94779 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d78c02773e454ea0b79cf1282eda125b. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:17.131978 11749 consensus_queue.cc:1035] T d78c02773e454ea0b79cf1282eda125b P 201d2b6485084e7dad2cbb722af94779 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:46:17.133044 6967 raft_consensus.cc:1261] T d78c02773e454ea0b79cf1282eda125b P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Refusing update from remote peer 201d2b6485084e7dad2cbb722af94779: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:17.133404 11749 consensus_queue.cc:1035] T d78c02773e454ea0b79cf1282eda125b P 201d2b6485084e7dad2cbb722af94779 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:46:17.134558 11767 raft_consensus.cc:2932] T d78c02773e454ea0b79cf1282eda125b P 201d2b6485084e7dad2cbb722af94779 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } } }
I20240502 20:46:17.134967 4374 raft_consensus.cc:2932] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } } }
I20240502 20:46:17.137991 4233 catalog_manager.cc:5484] T d78c02773e454ea0b79cf1282eda125b P 201d2b6485084e7dad2cbb722af94779 reported cstate change: config changed from index -1 to 2, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New cstate: current_term: 1 leader_uuid: "201d2b6485084e7dad2cbb722af94779" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:17.139405 6967 raft_consensus.cc:2932] T d78c02773e454ea0b79cf1282eda125b P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } } }
I20240502 20:46:17.146612 7099 consensus_queue.cc:237] T b69500b0144a42f59383b8c5d38324b3 P 201d2b6485084e7dad2cbb722af94779 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: true } }
I20240502 20:46:17.148144 6967 raft_consensus.cc:1261] T b69500b0144a42f59383b8c5d38324b3 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Refusing update from remote peer 201d2b6485084e7dad2cbb722af94779: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:17.148190 6702 raft_consensus.cc:1261] T b69500b0144a42f59383b8c5d38324b3 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Refusing update from remote peer 201d2b6485084e7dad2cbb722af94779: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:17.148566 11550 consensus_queue.cc:1035] T b69500b0144a42f59383b8c5d38324b3 P 201d2b6485084e7dad2cbb722af94779 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 20:46:17.148728 7035 consensus_peers.cc:485] T b69500b0144a42f59383b8c5d38324b3 P 201d2b6485084e7dad2cbb722af94779 -> Peer ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193): Couldn't send request to peer ed47c023452a497192f35d2f617bf9bd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b69500b0144a42f59383b8c5d38324b3. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:17.148880 11749 consensus_queue.cc:1035] T b69500b0144a42f59383b8c5d38324b3 P 201d2b6485084e7dad2cbb722af94779 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:46:17.150601 11550 raft_consensus.cc:2932] T b69500b0144a42f59383b8c5d38324b3 P 201d2b6485084e7dad2cbb722af94779 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: true } } }
I20240502 20:46:17.151217 6967 raft_consensus.cc:2932] T b69500b0144a42f59383b8c5d38324b3 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: true } } }
I20240502 20:46:17.151394 6702 raft_consensus.cc:2932] T b69500b0144a42f59383b8c5d38324b3 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: true } } }
I20240502 20:46:17.153981 4230 catalog_manager.cc:5484] T b69500b0144a42f59383b8c5d38324b3 P 201d2b6485084e7dad2cbb722af94779 reported cstate change: config changed from index -1 to 2, NON_VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) added. New cstate: current_term: 1 leader_uuid: "201d2b6485084e7dad2cbb722af94779" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:17.162070 6967 consensus_queue.cc:237] T ef4002126e8847809add182f262d62d9 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true } }
I20240502 20:46:17.163480 4768 raft_consensus.cc:1261] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Refusing update from remote peer a47e5c56f13843018ac584914a8d4707: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:17.163609 6835 raft_consensus.cc:1261] T ef4002126e8847809add182f262d62d9 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Refusing update from remote peer a47e5c56f13843018ac584914a8d4707: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 20:46:17.163746 6903 consensus_peers.cc:485] T ef4002126e8847809add182f262d62d9 P a47e5c56f13843018ac584914a8d4707 -> Peer c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827): Couldn't send request to peer c92e6e46782849479c8a0c6d82276fb2. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:17.163977 11798 consensus_queue.cc:1035] T ef4002126e8847809add182f262d62d9 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:17.164187 10138 consensus_queue.cc:1035] T ef4002126e8847809add182f262d62d9 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:17.166028 9156 raft_consensus.cc:2932] T ef4002126e8847809add182f262d62d9 P a47e5c56f13843018ac584914a8d4707 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true } } }
I20240502 20:46:17.166499 4768 raft_consensus.cc:2932] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true } } }
I20240502 20:46:17.167881 6835 raft_consensus.cc:2932] T ef4002126e8847809add182f262d62d9 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true } } }
I20240502 20:46:17.169044 4233 catalog_manager.cc:5484] T ef4002126e8847809add182f262d62d9 P a47e5c56f13843018ac584914a8d4707 reported cstate change: config changed from index 4 to 5, NON_VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) added. New cstate: current_term: 1 leader_uuid: "a47e5c56f13843018ac584914a8d4707" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:17.178282 6967 consensus_queue.cc:237] T a999f994b78a4b33a95ee7d5407cba20 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } }
I20240502 20:46:17.179715 6702 raft_consensus.cc:1261] T a999f994b78a4b33a95ee7d5407cba20 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Refusing update from remote peer a47e5c56f13843018ac584914a8d4707: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:17.179811 6835 raft_consensus.cc:1261] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Refusing update from remote peer a47e5c56f13843018ac584914a8d4707: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 20:46:17.179852 6902 consensus_peers.cc:485] T a999f994b78a4b33a95ee7d5407cba20 P a47e5c56f13843018ac584914a8d4707 -> Peer c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097): Couldn't send request to peer c3d26eafd51547acb780d4aac5ebe7cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a999f994b78a4b33a95ee7d5407cba20. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:17.180105 10138 consensus_queue.cc:1035] T a999f994b78a4b33a95ee7d5407cba20 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:46:17.180306 9156 consensus_queue.cc:1035] T a999f994b78a4b33a95ee7d5407cba20 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:46:17.182101 10138 raft_consensus.cc:2932] T a999f994b78a4b33a95ee7d5407cba20 P a47e5c56f13843018ac584914a8d4707 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } } }
I20240502 20:46:17.182458 6702 raft_consensus.cc:2932] T a999f994b78a4b33a95ee7d5407cba20 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } } }
I20240502 20:46:17.183694 6831 raft_consensus.cc:2932] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } } }
I20240502 20:46:17.185153 4231 catalog_manager.cc:5484] T a999f994b78a4b33a95ee7d5407cba20 P 7845c397230d49358ba2fad9db53f97c reported cstate change: config changed from index -1 to 2, NON_VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) added. New cstate: current_term: 1 leader_uuid: "a47e5c56f13843018ac584914a8d4707" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } } }
I20240502 20:46:17.188351 4235 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41812:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 20:46:17.193825 6802 tablet_service.cc:1467] Processing CreateTablet for tablet 2a44b2af7886427db821f873e5a79151 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c863a144af7347ea87a5e829dd2608be]), partition=
I20240502 20:46:17.194033 7079 tablet_service.cc:1467] Processing CreateTablet for tablet 2a44b2af7886427db821f873e5a79151 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c863a144af7347ea87a5e829dd2608be]), partition=
I20240502 20:46:17.194101 6802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a44b2af7886427db821f873e5a79151. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:17.194281 7079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a44b2af7886427db821f873e5a79151. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:17.196352 11736 tablet_bootstrap.cc:492] T 2a44b2af7886427db821f873e5a79151 P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap starting.
I20240502 20:46:17.197194 6947 tablet_service.cc:1467] Processing CreateTablet for tablet 2a44b2af7886427db821f873e5a79151 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c863a144af7347ea87a5e829dd2608be]), partition=
I20240502 20:46:17.197415 6947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a44b2af7886427db821f873e5a79151. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:17.197579 11736 tablet_bootstrap.cc:654] T 2a44b2af7886427db821f873e5a79151 P a4bf1df7fa6b4948be502c182bfbb833: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:17.198074 4638 consensus_queue.cc:237] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } }
I20240502 20:46:17.198421 11814 tablet_bootstrap.cc:492] T 2a44b2af7886427db821f873e5a79151 P 201d2b6485084e7dad2cbb722af94779: Bootstrap starting.
I20240502 20:46:17.199795 11736 tablet_bootstrap.cc:492] T 2a44b2af7886427db821f873e5a79151 P a4bf1df7fa6b4948be502c182bfbb833: No bootstrap required, opened a new log
I20240502 20:46:17.199887 11736 ts_tablet_manager.cc:1367] T 2a44b2af7886427db821f873e5a79151 P a4bf1df7fa6b4948be502c182bfbb833: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 20:46:17.200065 4374 raft_consensus.cc:1261] T 253d158b340a444182793f18239bfa22 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:17.200093 4507 raft_consensus.cc:1261] T 253d158b340a444182793f18239bfa22 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:17.200182 11814 tablet_bootstrap.cc:654] T 2a44b2af7886427db821f873e5a79151 P 201d2b6485084e7dad2cbb722af94779: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:17.200405 11736 raft_consensus.cc:348] T 2a44b2af7886427db821f873e5a79151 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:17.200505 11736 raft_consensus.cc:374] T 2a44b2af7886427db821f873e5a79151 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:17.200445 9041 consensus_queue.cc:1035] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:17.200533 11736 raft_consensus.cc:729] T 2a44b2af7886427db821f873e5a79151 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Initialized, Role: FOLLOWER
I20240502 20:46:17.200640 11736 consensus_queue.cc:260] T 2a44b2af7886427db821f873e5a79151 P a4bf1df7fa6b4948be502c182bfbb833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:17.200922 11736 ts_tablet_manager.cc:1398] T 2a44b2af7886427db821f873e5a79151 P a4bf1df7fa6b4948be502c182bfbb833: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 20:46:17.201148 11732 tablet_bootstrap.cc:492] T 2a44b2af7886427db821f873e5a79151 P a47e5c56f13843018ac584914a8d4707: Bootstrap starting.
I20240502 20:46:17.201588 11814 tablet_bootstrap.cc:492] T 2a44b2af7886427db821f873e5a79151 P 201d2b6485084e7dad2cbb722af94779: No bootstrap required, opened a new log
I20240502 20:46:17.201675 11814 ts_tablet_manager.cc:1367] T 2a44b2af7886427db821f873e5a79151 P 201d2b6485084e7dad2cbb722af94779: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:46:17.201990 11732 tablet_bootstrap.cc:654] T 2a44b2af7886427db821f873e5a79151 P a47e5c56f13843018ac584914a8d4707: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:17.202118 11814 raft_consensus.cc:348] T 2a44b2af7886427db821f873e5a79151 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:17.202229 11814 raft_consensus.cc:374] T 2a44b2af7886427db821f873e5a79151 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:17.202263 11814 raft_consensus.cc:729] T 2a44b2af7886427db821f873e5a79151 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 201d2b6485084e7dad2cbb722af94779, State: Initialized, Role: FOLLOWER
I20240502 20:46:17.202379 11814 consensus_queue.cc:260] T 2a44b2af7886427db821f873e5a79151 P 201d2b6485084e7dad2cbb722af94779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:17.202682 11814 ts_tablet_manager.cc:1398] T 2a44b2af7886427db821f873e5a79151 P 201d2b6485084e7dad2cbb722af94779: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240502 20:46:17.202893 4572 consensus_peers.cc:485] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 253d158b340a444182793f18239bfa22. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:17.202988 9041 consensus_queue.cc:1035] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:17.204072 9041 raft_consensus.cc:2932] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } } }
I20240502 20:46:17.205343 4374 raft_consensus.cc:2932] T 253d158b340a444182793f18239bfa22 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } } }
I20240502 20:46:17.205827 11732 tablet_bootstrap.cc:492] T 2a44b2af7886427db821f873e5a79151 P a47e5c56f13843018ac584914a8d4707: No bootstrap required, opened a new log
I20240502 20:46:17.205914 11732 ts_tablet_manager.cc:1367] T 2a44b2af7886427db821f873e5a79151 P a47e5c56f13843018ac584914a8d4707: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 20:46:17.206383 11732 raft_consensus.cc:348] T 2a44b2af7886427db821f873e5a79151 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:17.206478 11732 raft_consensus.cc:374] T 2a44b2af7886427db821f873e5a79151 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:17.206507 11732 raft_consensus.cc:729] T 2a44b2af7886427db821f873e5a79151 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Initialized, Role: FOLLOWER
I20240502 20:46:17.206637 11732 consensus_queue.cc:260] T 2a44b2af7886427db821f873e5a79151 P a47e5c56f13843018ac584914a8d4707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:17.206897 11732 ts_tablet_manager.cc:1398] T 2a44b2af7886427db821f873e5a79151 P a47e5c56f13843018ac584914a8d4707: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:17.209930 4233 catalog_manager.cc:5484] T 253d158b340a444182793f18239bfa22 P c92e6e46782849479c8a0c6d82276fb2 reported cstate change: config changed from index -1 to 4, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New cstate: current_term: 1 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } } }
I20240502 20:46:17.215535 4507 raft_consensus.cc:2932] T 253d158b340a444182793f18239bfa22 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: NON_VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { promote: true } } }
I20240502 20:46:17.220290 6702 consensus_queue.cc:237] T 3d938d01d5154590a9262acd54094ad2 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: true } }
W20240502 20:46:17.221907 6639 consensus_peers.cc:485] T 3d938d01d5154590a9262acd54094ad2 P 7845c397230d49358ba2fad9db53f97c -> Peer ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193): Couldn't send request to peer ed47c023452a497192f35d2f617bf9bd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3d938d01d5154590a9262acd54094ad2. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:17.221940 7099 raft_consensus.cc:1261] T 3d938d01d5154590a9262acd54094ad2 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Refusing update from remote peer 7845c397230d49358ba2fad9db53f97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:17.221940 4638 raft_consensus.cc:1261] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer 7845c397230d49358ba2fad9db53f97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:17.222332 10922 consensus_queue.cc:1035] T 3d938d01d5154590a9262acd54094ad2 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:17.222508 10504 consensus_queue.cc:1035] T 3d938d01d5154590a9262acd54094ad2 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:17.223784 11605 raft_consensus.cc:482] T dff789a7a50c4ff0a1eb29489ac7901a P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:17.223896 11605 raft_consensus.cc:504] T dff789a7a50c4ff0a1eb29489ac7901a P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:17.224305 11605 leader_election.cc:288] T dff789a7a50c4ff0a1eb29489ac7901a P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097)
I20240502 20:46:17.224319 11746 raft_consensus.cc:2932] T 3d938d01d5154590a9262acd54094ad2 P 7845c397230d49358ba2fad9db53f97c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: true } } }
I20240502 20:46:17.224756 4638 raft_consensus.cc:2932] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: true } } }
I20240502 20:46:17.225235 6702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dff789a7a50c4ff0a1eb29489ac7901a" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7845c397230d49358ba2fad9db53f97c" is_pre_election: true
I20240502 20:46:17.225363 6702 raft_consensus.cc:2454] T dff789a7a50c4ff0a1eb29489ac7901a P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 0.
I20240502 20:46:17.225629 6770 leader_election.cc:302] T dff789a7a50c4ff0a1eb29489ac7901a P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7845c397230d49358ba2fad9db53f97c, a4bf1df7fa6b4948be502c182bfbb833; no voters:
I20240502 20:46:17.225786 11605 raft_consensus.cc:2781] T dff789a7a50c4ff0a1eb29489ac7901a P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:17.225844 11605 raft_consensus.cc:482] T dff789a7a50c4ff0a1eb29489ac7901a P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:17.225889 11605 raft_consensus.cc:3037] T dff789a7a50c4ff0a1eb29489ac7901a P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:17.226291 4507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dff789a7a50c4ff0a1eb29489ac7901a" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" is_pre_election: true
I20240502 20:46:17.226410 4507 raft_consensus.cc:2454] T dff789a7a50c4ff0a1eb29489ac7901a P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 0.
I20240502 20:46:17.226925 11605 raft_consensus.cc:504] T dff789a7a50c4ff0a1eb29489ac7901a P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:17.227294 11605 leader_election.cc:288] T dff789a7a50c4ff0a1eb29489ac7901a P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 election: Requested vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097)
I20240502 20:46:17.227650 4507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dff789a7a50c4ff0a1eb29489ac7901a" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc"
I20240502 20:46:17.227778 4507 raft_consensus.cc:3037] T dff789a7a50c4ff0a1eb29489ac7901a P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:17.227598 7099 raft_consensus.cc:2932] T 3d938d01d5154590a9262acd54094ad2 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: true } } }
I20240502 20:46:17.228037 6702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dff789a7a50c4ff0a1eb29489ac7901a" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7845c397230d49358ba2fad9db53f97c"
I20240502 20:46:17.228130 6702 raft_consensus.cc:3037] T dff789a7a50c4ff0a1eb29489ac7901a P 7845c397230d49358ba2fad9db53f97c [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:17.228823 4507 raft_consensus.cc:2454] T dff789a7a50c4ff0a1eb29489ac7901a P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 1.
I20240502 20:46:17.229046 6702 raft_consensus.cc:2454] T dff789a7a50c4ff0a1eb29489ac7901a P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 1.
I20240502 20:46:17.229411 6770 leader_election.cc:302] T dff789a7a50c4ff0a1eb29489ac7901a P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7845c397230d49358ba2fad9db53f97c, a4bf1df7fa6b4948be502c182bfbb833; no voters:
I20240502 20:46:17.230716 11605 raft_consensus.cc:2781] T dff789a7a50c4ff0a1eb29489ac7901a P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:17.230805 11605 raft_consensus.cc:686] T dff789a7a50c4ff0a1eb29489ac7901a P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEADER]: Becoming Leader. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Running, Role: LEADER
I20240502 20:46:17.230943 11605 consensus_queue.cc:237] T dff789a7a50c4ff0a1eb29489ac7901a P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:17.230412 4233 catalog_manager.cc:5484] T 3d938d01d5154590a9262acd54094ad2 P 7845c397230d49358ba2fad9db53f97c reported cstate change: config changed from index 4 to 5, NON_VOTER ed47c023452a497192f35d2f617bf9bd (127.30.32.68) added. New cstate: current_term: 1 leader_uuid: "7845c397230d49358ba2fad9db53f97c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:17.233794 4235 catalog_manager.cc:5484] T dff789a7a50c4ff0a1eb29489ac7901a P a4bf1df7fa6b4948be502c182bfbb833 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70). New cstate: current_term: 1 leader_uuid: "a4bf1df7fa6b4948be502c182bfbb833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:17.243465 6702 consensus_queue.cc:237] T 533e7f3c852c4fd9bd5093085a5e8e36 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true } }
I20240502 20:46:17.244804 7099 raft_consensus.cc:1261] T 533e7f3c852c4fd9bd5093085a5e8e36 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Refusing update from remote peer 7845c397230d49358ba2fad9db53f97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:17.244849 6967 raft_consensus.cc:1261] T 533e7f3c852c4fd9bd5093085a5e8e36 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Refusing update from remote peer 7845c397230d49358ba2fad9db53f97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:17.245211 10922 consensus_queue.cc:1035] T 533e7f3c852c4fd9bd5093085a5e8e36 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:46:17.245844 10504 consensus_queue.cc:1035] T 533e7f3c852c4fd9bd5093085a5e8e36 P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 20:46:17.246935 6639 consensus_peers.cc:485] T 533e7f3c852c4fd9bd5093085a5e8e36 P 7845c397230d49358ba2fad9db53f97c -> Peer c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827): Couldn't send request to peer c92e6e46782849479c8a0c6d82276fb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 533e7f3c852c4fd9bd5093085a5e8e36. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:17.247565 10504 raft_consensus.cc:2932] T 533e7f3c852c4fd9bd5093085a5e8e36 P 7845c397230d49358ba2fad9db53f97c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true } } }
I20240502 20:46:17.247907 6967 raft_consensus.cc:2932] T 533e7f3c852c4fd9bd5093085a5e8e36 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true } } }
I20240502 20:46:17.250381 7099 raft_consensus.cc:2932] T 533e7f3c852c4fd9bd5093085a5e8e36 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true } } }
I20240502 20:46:17.250900 4235 catalog_manager.cc:5484] T 533e7f3c852c4fd9bd5093085a5e8e36 P a47e5c56f13843018ac584914a8d4707 reported cstate change: config changed from index -1 to 2, NON_VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) added. New cstate: current_term: 1 leader_uuid: "7845c397230d49358ba2fad9db53f97c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true } } }
I20240502 20:46:17.266101 4768 consensus_queue.cc:237] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } }
I20240502 20:46:17.267637 7099 raft_consensus.cc:1261] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:17.267814 6831 raft_consensus.cc:1261] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 20:46:17.268260 4703 consensus_peers.cc:485] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P ed47c023452a497192f35d2f617bf9bd -> Peer 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141): Couldn't send request to peer 923cfab926e7443f8ffd7b21ad0a9d91. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 77b294bcc497479e8e8bd8e7f9cc1ff2. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:17.268308 11847 consensus_queue.cc:1035] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:46:17.268477 9125 consensus_queue.cc:1035] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:46:17.270174 9125 raft_consensus.cc:2932] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } } }
I20240502 20:46:17.270692 7099 raft_consensus.cc:2932] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } } }
I20240502 20:46:17.270717 6831 raft_consensus.cc:2932] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } } }
I20240502 20:46:17.274478 4233 catalog_manager.cc:5484] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P a4bf1df7fa6b4948be502c182bfbb833 reported cstate change: config changed from index -1 to 2, NON_VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) added. New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } } }
I20240502 20:46:17.282050 11605 raft_consensus.cc:482] T 172976cb61a649d3995425d9859b432c P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:17.282142 11605 raft_consensus.cc:504] T 172976cb61a649d3995425d9859b432c P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } }
I20240502 20:46:17.282572 11605 leader_election.cc:288] T 172976cb61a649d3995425d9859b432c P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141), c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
I20240502 20:46:17.283349 4374 consensus_queue.cc:237] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } }
I20240502 20:46:17.283372 4638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "172976cb61a649d3995425d9859b432c" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" is_pre_election: true
I20240502 20:46:17.283569 4638 raft_consensus.cc:2379] T 172976cb61a649d3995425d9859b432c P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4bf1df7fa6b4948be502c182bfbb833 in current term 1: Already voted for candidate c92e6e46782849479c8a0c6d82276fb2 in this term.
I20240502 20:46:17.284801 6702 raft_consensus.cc:1261] T cedd1c79256a4f32a73b6a88244e6181 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 20:46:17.285071 4313 consensus_peers.cc:485] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cedd1c79256a4f32a73b6a88244e6181. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:17.285252 6831 raft_consensus.cc:1261] T cedd1c79256a4f32a73b6a88244e6181 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Refusing update from remote peer c92e6e46782849479c8a0c6d82276fb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:17.285214 11070 consensus_queue.cc:1035] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:46:17.285684 11070 consensus_queue.cc:1035] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:46:17.287406 4374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "172976cb61a649d3995425d9859b432c" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2" is_pre_election: true
I20240502 20:46:17.287752 6771 leader_election.cc:302] T 172976cb61a649d3995425d9859b432c P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4bf1df7fa6b4948be502c182bfbb833; no voters: 923cfab926e7443f8ffd7b21ad0a9d91, c92e6e46782849479c8a0c6d82276fb2
I20240502 20:46:17.287415 11070 raft_consensus.cc:2932] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } } }
I20240502 20:46:17.287767 6702 raft_consensus.cc:2932] T cedd1c79256a4f32a73b6a88244e6181 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } } }
I20240502 20:46:17.287981 11605 raft_consensus.cc:3037] T 172976cb61a649d3995425d9859b432c P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:17.288769 11605 raft_consensus.cc:2726] T 172976cb61a649d3995425d9859b432c P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 20:46:17.289080 6831 raft_consensus.cc:2932] T cedd1c79256a4f32a73b6a88244e6181 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } } }
I20240502 20:46:17.290987 4231 catalog_manager.cc:5484] T cedd1c79256a4f32a73b6a88244e6181 P 7845c397230d49358ba2fad9db53f97c reported cstate change: config changed from index -1 to 2, NON_VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) added. New cstate: current_term: 1 leader_uuid: "c92e6e46782849479c8a0c6d82276fb2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: NON_VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: true } } }
I20240502 20:46:17.299367 6831 consensus_queue.cc:237] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } }
I20240502 20:46:17.300922 6967 raft_consensus.cc:1261] T b2508a5aa1c14b63971895953e149e17 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Refusing update from remote peer a4bf1df7fa6b4948be502c182bfbb833: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:17.301350 11605 consensus_queue.cc:1035] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 20:46:17.301542 6770 consensus_peers.cc:485] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097): Couldn't send request to peer c3d26eafd51547acb780d4aac5ebe7cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b2508a5aa1c14b63971895953e149e17. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:17.301556 7099 raft_consensus.cc:1261] T b2508a5aa1c14b63971895953e149e17 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Refusing update from remote peer a4bf1df7fa6b4948be502c182bfbb833: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:17.301971 11605 consensus_queue.cc:1035] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:46:17.303488 11863 raft_consensus.cc:2932] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } } }
I20240502 20:46:17.304044 6967 raft_consensus.cc:2932] T b2508a5aa1c14b63971895953e149e17 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } } }
I20240502 20:46:17.304055 7099 raft_consensus.cc:2932] T b2508a5aa1c14b63971895953e149e17 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } } }
I20240502 20:46:17.306973 4233 catalog_manager.cc:5484] T b2508a5aa1c14b63971895953e149e17 P a47e5c56f13843018ac584914a8d4707 reported cstate change: config changed from index -1 to 2, NON_VOTER c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66) added. New cstate: current_term: 1 leader_uuid: "a4bf1df7fa6b4948be502c182bfbb833" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } } }
I20240502 20:46:17.314167 4768 consensus_queue.cc:237] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } }
I20240502 20:46:17.315775 6967 raft_consensus.cc:1261] T 73efeba3d3094ef387e4efb03cff9d4e P a47e5c56f13843018ac584914a8d4707 [term 2 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 20:46:17.315861 4374 raft_consensus.cc:1261] T 73efeba3d3094ef387e4efb03cff9d4e P c92e6e46782849479c8a0c6d82276fb2 [term 2 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
W20240502 20:46:17.316131 4705 consensus_peers.cc:485] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 73efeba3d3094ef387e4efb03cff9d4e. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:17.316236 11847 consensus_queue.cc:1035] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:17.316782 11847 consensus_queue.cc:1035] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:17.318306 9125 raft_consensus.cc:2932] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } } }
I20240502 20:46:17.319278 6967 raft_consensus.cc:2932] T 73efeba3d3094ef387e4efb03cff9d4e P a47e5c56f13843018ac584914a8d4707 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } } }
I20240502 20:46:17.320101 4377 raft_consensus.cc:2932] T 73efeba3d3094ef387e4efb03cff9d4e P c92e6e46782849479c8a0c6d82276fb2 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } } }
I20240502 20:46:17.322351 4231 catalog_manager.cc:5484] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd reported cstate change: config changed from index 7 to 8, NON_VOTER 7845c397230d49358ba2fad9db53f97c (127.30.32.69) added. New cstate: current_term: 2 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:17.330401 4768 consensus_queue.cc:237] T 3ba2f5d327f3403ea1604aae9485b960 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } }
I20240502 20:46:17.331540 6967 raft_consensus.cc:1261] T 3ba2f5d327f3403ea1604aae9485b960 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:17.331712 4377 raft_consensus.cc:1261] T 3ba2f5d327f3403ea1604aae9485b960 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 20:46:17.331689 4703 consensus_peers.cc:485] T 3ba2f5d327f3403ea1604aae9485b960 P ed47c023452a497192f35d2f617bf9bd -> Peer 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141): Couldn't send request to peer 923cfab926e7443f8ffd7b21ad0a9d91. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3ba2f5d327f3403ea1604aae9485b960. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:17.331907 9125 consensus_queue.cc:1035] T 3ba2f5d327f3403ea1604aae9485b960 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:46:17.332091 11416 consensus_queue.cc:1035] T 3ba2f5d327f3403ea1604aae9485b960 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:46:17.333892 11416 raft_consensus.cc:2932] T 3ba2f5d327f3403ea1604aae9485b960 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } } }
I20240502 20:46:17.334313 4377 raft_consensus.cc:2932] T 3ba2f5d327f3403ea1604aae9485b960 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } } }
I20240502 20:46:17.334347 6967 raft_consensus.cc:2932] T 3ba2f5d327f3403ea1604aae9485b960 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } } }
I20240502 20:46:17.337448 4231 catalog_manager.cc:5484] T 3ba2f5d327f3403ea1604aae9485b960 P ed47c023452a497192f35d2f617bf9bd reported cstate change: config changed from index -1 to 2, NON_VOTER 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) added. New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:17.345213 6967 consensus_queue.cc:237] T 8602022a45e1482ebf8dab9930ce8209 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true } }
I20240502 20:46:17.346594 4638 raft_consensus.cc:1261] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer a47e5c56f13843018ac584914a8d4707: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:17.346669 6702 raft_consensus.cc:1261] T 8602022a45e1482ebf8dab9930ce8209 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Refusing update from remote peer a47e5c56f13843018ac584914a8d4707: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 20:46:17.346725 6903 consensus_peers.cc:485] T 8602022a45e1482ebf8dab9930ce8209 P a47e5c56f13843018ac584914a8d4707 -> Peer c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827): Couldn't send request to peer c92e6e46782849479c8a0c6d82276fb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8602022a45e1482ebf8dab9930ce8209. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:17.346998 11798 consensus_queue.cc:1035] T 8602022a45e1482ebf8dab9930ce8209 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:17.347430 11798 consensus_queue.cc:1035] T 8602022a45e1482ebf8dab9930ce8209 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:17.348959 9156 raft_consensus.cc:2932] T 8602022a45e1482ebf8dab9930ce8209 P a47e5c56f13843018ac584914a8d4707 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true } } }
I20240502 20:46:17.349548 6702 raft_consensus.cc:2932] T 8602022a45e1482ebf8dab9930ce8209 P 7845c397230d49358ba2fad9db53f97c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true } } }
I20240502 20:46:17.349548 4638 raft_consensus.cc:2932] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true } } }
I20240502 20:46:17.352996 4230 catalog_manager.cc:5484] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index 4 to 5, NON_VOTER c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) added. New cstate: current_term: 1 leader_uuid: "a47e5c56f13843018ac584914a8d4707" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true } } }
I20240502 20:46:17.372232 11798 raft_consensus.cc:482] T 2a44b2af7886427db821f873e5a79151 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:17.372378 11798 raft_consensus.cc:504] T 2a44b2af7886427db821f873e5a79151 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:17.372826 11798 leader_election.cc:288] T 2a44b2af7886427db821f873e5a79151 P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
I20240502 20:46:17.373171 6831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a44b2af7886427db821f873e5a79151" candidate_uuid: "a47e5c56f13843018ac584914a8d4707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833" is_pre_election: true
I20240502 20:46:17.373221 7099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a44b2af7886427db821f873e5a79151" candidate_uuid: "a47e5c56f13843018ac584914a8d4707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201d2b6485084e7dad2cbb722af94779" is_pre_election: true
I20240502 20:46:17.373343 7099 raft_consensus.cc:2454] T 2a44b2af7886427db821f873e5a79151 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a47e5c56f13843018ac584914a8d4707 in term 0.
I20240502 20:46:17.373323 6831 raft_consensus.cc:2454] T 2a44b2af7886427db821f873e5a79151 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a47e5c56f13843018ac584914a8d4707 in term 0.
I20240502 20:46:17.373674 6901 leader_election.cc:302] T 2a44b2af7886427db821f873e5a79151 P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a47e5c56f13843018ac584914a8d4707, a4bf1df7fa6b4948be502c182bfbb833; no voters:
I20240502 20:46:17.373858 11798 raft_consensus.cc:2781] T 2a44b2af7886427db821f873e5a79151 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:17.373941 11798 raft_consensus.cc:482] T 2a44b2af7886427db821f873e5a79151 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:17.373994 11798 raft_consensus.cc:3037] T 2a44b2af7886427db821f873e5a79151 P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:17.375171 11798 raft_consensus.cc:504] T 2a44b2af7886427db821f873e5a79151 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:17.375634 11798 leader_election.cc:288] T 2a44b2af7886427db821f873e5a79151 P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 1 election: Requested vote from peers a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
I20240502 20:46:17.375936 6831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a44b2af7886427db821f873e5a79151" candidate_uuid: "a47e5c56f13843018ac584914a8d4707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833"
I20240502 20:46:17.376056 6831 raft_consensus.cc:3037] T 2a44b2af7886427db821f873e5a79151 P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:17.377125 7099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a44b2af7886427db821f873e5a79151" candidate_uuid: "a47e5c56f13843018ac584914a8d4707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201d2b6485084e7dad2cbb722af94779"
I20240502 20:46:17.377255 7099 raft_consensus.cc:3037] T 2a44b2af7886427db821f873e5a79151 P 201d2b6485084e7dad2cbb722af94779 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:17.378299 7099 raft_consensus.cc:2454] T 2a44b2af7886427db821f873e5a79151 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a47e5c56f13843018ac584914a8d4707 in term 1.
I20240502 20:46:17.378564 6900 leader_election.cc:302] T 2a44b2af7886427db821f873e5a79151 P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201d2b6485084e7dad2cbb722af94779, a47e5c56f13843018ac584914a8d4707; no voters:
I20240502 20:46:17.378747 11798 raft_consensus.cc:2781] T 2a44b2af7886427db821f873e5a79151 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:17.378842 11798 raft_consensus.cc:686] T 2a44b2af7886427db821f873e5a79151 P a47e5c56f13843018ac584914a8d4707 [term 1 LEADER]: Becoming Leader. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Running, Role: LEADER
I20240502 20:46:17.378962 6831 raft_consensus.cc:2454] T 2a44b2af7886427db821f873e5a79151 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a47e5c56f13843018ac584914a8d4707 in term 1.
I20240502 20:46:17.378988 11798 consensus_queue.cc:237] T 2a44b2af7886427db821f873e5a79151 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:17.381067 4233 catalog_manager.cc:5484] T 2a44b2af7886427db821f873e5a79151 P a47e5c56f13843018ac584914a8d4707 reported cstate change: term changed from 0 to 1, leader changed from <none> to a47e5c56f13843018ac584914a8d4707 (127.30.32.71). New cstate: current_term: 1 leader_uuid: "a47e5c56f13843018ac584914a8d4707" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: HEALTHY } } }
I20240502 20:46:17.433059 10922 consensus_queue.cc:1035] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:17.436235 10922 consensus_queue.cc:1035] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P 7845c397230d49358ba2fad9db53f97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:17.484884 11608 consensus_queue.cc:1035] T 172976cb61a649d3995425d9859b432c P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:17.488966 11608 consensus_queue.cc:1035] T 172976cb61a649d3995425d9859b432c P c92e6e46782849479c8a0c6d82276fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:17.505908 4230 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 20:46:17.510035 7099 raft_consensus.cc:1261] T 2a44b2af7886427db821f873e5a79151 P 201d2b6485084e7dad2cbb722af94779 [term 1 FOLLOWER]: Refusing update from remote peer a47e5c56f13843018ac584914a8d4707: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:17.510042 6835 raft_consensus.cc:1261] T 2a44b2af7886427db821f873e5a79151 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Refusing update from remote peer a47e5c56f13843018ac584914a8d4707: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:17.510413 11798 consensus_queue.cc:1035] T 2a44b2af7886427db821f873e5a79151 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:17.510560 10138 consensus_queue.cc:1035] T 2a44b2af7886427db821f873e5a79151 P a47e5c56f13843018ac584914a8d4707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:17.513221 4230 catalog_manager.cc:5576] TS a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867) does not have the latest schema for tablet 2a44b2af7886427db821f873e5a79151 (table rebalancer_dynamic_table [id=c863a144af7347ea87a5e829dd2608be]). Expected version 1 got 0
I20240502 20:46:17.513839 4233 catalog_manager.cc:5576] TS 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643) does not have the latest schema for tablet 2a44b2af7886427db821f873e5a79151 (table rebalancer_dynamic_table [id=c863a144af7347ea87a5e829dd2608be]). Expected version 1 got 0
I20240502 20:46:17.514288 11983 tablet.cc:1722] T 2a44b2af7886427db821f873e5a79151 P a47e5c56f13843018ac584914a8d4707: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 20:46:17.515540 4233 catalog_manager.cc:5576] TS a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841) does not have the latest schema for tablet 2a44b2af7886427db821f873e5a79151 (table rebalancer_dynamic_table [id=c863a144af7347ea87a5e829dd2608be]). Expected version 1 got 0
I20240502 20:46:17.514591 11983 tablet.cc:1620] T 2a44b2af7886427db821f873e5a79151 P a47e5c56f13843018ac584914a8d4707: MemRowSet was empty: no flush needed.
I20240502 20:46:17.515833 11984 tablet.cc:1722] T 2a44b2af7886427db821f873e5a79151 P 201d2b6485084e7dad2cbb722af94779: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 20:46:17.516120 11984 tablet.cc:1620] T 2a44b2af7886427db821f873e5a79151 P 201d2b6485084e7dad2cbb722af94779: MemRowSet was empty: no flush needed.
I20240502 20:46:17.517045 11985 tablet.cc:1722] T 2a44b2af7886427db821f873e5a79151 P a4bf1df7fa6b4948be502c182bfbb833: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 20:46:17.517359 11985 tablet.cc:1620] T 2a44b2af7886427db821f873e5a79151 P a4bf1df7fa6b4948be502c182bfbb833: MemRowSet was empty: no flush needed.
I20240502 20:46:17.520110 4233 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: rebalancer_dynamic_table [id=c863a144af7347ea87a5e829dd2608be] alter complete (version 1)
I20240502 20:46:17.552534 11986 ts_tablet_manager.cc:902] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P 923cfab926e7443f8ffd7b21ad0a9d91: Initiating tablet copy from peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
I20240502 20:46:17.553150 11986 tablet_copy_client.cc:308] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P 923cfab926e7443f8ffd7b21ad0a9d91: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.69:42429
I20240502 20:46:17.553670 6723 tablet_copy_service.cc:140] P 7845c397230d49358ba2fad9db53f97c: Received BeginTabletCopySession request for tablet 75e2e48cd7a4497f9ccaae4f712f3ef0 from peer 923cfab926e7443f8ffd7b21ad0a9d91 ({username='slave'} at 127.30.32.67:41973)
I20240502 20:46:17.553810 6723 tablet_copy_service.cc:161] P 7845c397230d49358ba2fad9db53f97c: Beginning new tablet copy session on tablet 75e2e48cd7a4497f9ccaae4f712f3ef0 from peer 923cfab926e7443f8ffd7b21ad0a9d91 at {username='slave'} at 127.30.32.67:41973: session id = 923cfab926e7443f8ffd7b21ad0a9d91-75e2e48cd7a4497f9ccaae4f712f3ef0
I20240502 20:46:17.555131 6723 tablet_copy_source_session.cc:205] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P 7845c397230d49358ba2fad9db53f97c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:17.556183 11986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75e2e48cd7a4497f9ccaae4f712f3ef0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:17.559185 11986 tablet_copy_client.cc:783] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P 923cfab926e7443f8ffd7b21ad0a9d91: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:17.559334 11986 tablet_copy_client.cc:647] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P 923cfab926e7443f8ffd7b21ad0a9d91: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:17.560451 11986 tablet_copy_client.cc:523] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P 923cfab926e7443f8ffd7b21ad0a9d91: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:17.562039 11986 tablet_bootstrap.cc:492] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:46:17.566867 11986 tablet_bootstrap.cc:492] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:17.567158 11986 tablet_bootstrap.cc:492] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap complete.
I20240502 20:46:17.567341 11986 ts_tablet_manager.cc:1367] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.002s
I20240502 20:46:17.567909 11986 raft_consensus.cc:348] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } }
I20240502 20:46:17.568068 11986 raft_consensus.cc:729] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: LEARNER
I20240502 20:46:17.568192 11986 consensus_queue.cc:260] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } }
I20240502 20:46:17.568595 11986 ts_tablet_manager.cc:1398] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240502 20:46:17.569183 6723 tablet_copy_service.cc:342] P 7845c397230d49358ba2fad9db53f97c: Request end of tablet copy session 923cfab926e7443f8ffd7b21ad0a9d91-75e2e48cd7a4497f9ccaae4f712f3ef0 received from {username='slave'} at 127.30.32.67:41973
I20240502 20:46:17.569275 6723 tablet_copy_service.cc:433] P 7845c397230d49358ba2fad9db53f97c: ending tablet copy session 923cfab926e7443f8ffd7b21ad0a9d91-75e2e48cd7a4497f9ccaae4f712f3ef0 on tablet 75e2e48cd7a4497f9ccaae4f712f3ef0 with peer 923cfab926e7443f8ffd7b21ad0a9d91
I20240502 20:46:17.622359 4233 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 20:46:17.628053 11983 tablet.cc:1722] T 2a44b2af7886427db821f873e5a79151 P a47e5c56f13843018ac584914a8d4707: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 20:46:17.628537 11983 tablet.cc:1620] T 2a44b2af7886427db821f873e5a79151 P a47e5c56f13843018ac584914a8d4707: MemRowSet was empty: no flush needed.
I20240502 20:46:17.629472 11985 tablet.cc:1722] T 2a44b2af7886427db821f873e5a79151 P a4bf1df7fa6b4948be502c182bfbb833: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 20:46:17.629801 11985 tablet.cc:1620] T 2a44b2af7886427db821f873e5a79151 P a4bf1df7fa6b4948be502c182bfbb833: MemRowSet was empty: no flush needed.
I20240502 20:46:17.632242 11984 tablet.cc:1722] T 2a44b2af7886427db821f873e5a79151 P 201d2b6485084e7dad2cbb722af94779: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 20:46:17.632531 11984 tablet.cc:1620] T 2a44b2af7886427db821f873e5a79151 P 201d2b6485084e7dad2cbb722af94779: MemRowSet was empty: no flush needed.
I20240502 20:46:17.637773 4233 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: rebalancer_dynamic_table [id=c863a144af7347ea87a5e829dd2608be] alter complete (version 2)
I20240502 20:46:17.639345 11863 consensus_queue.cc:1035] T dff789a7a50c4ff0a1eb29489ac7901a P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:17.643661 11605 consensus_queue.cc:1035] T dff789a7a50c4ff0a1eb29489ac7901a P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:17.660907 12000 ts_tablet_manager.cc:902] T 6ff3c95ccfbf486fbc4bfc636afa829b P c92e6e46782849479c8a0c6d82276fb2: Initiating tablet copy from peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
I20240502 20:46:17.662902 12000 tablet_copy_client.cc:308] T 6ff3c95ccfbf486fbc4bfc636afa829b P c92e6e46782849479c8a0c6d82276fb2: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.69:42429
I20240502 20:46:17.663813 6723 tablet_copy_service.cc:140] P 7845c397230d49358ba2fad9db53f97c: Received BeginTabletCopySession request for tablet 6ff3c95ccfbf486fbc4bfc636afa829b from peer c92e6e46782849479c8a0c6d82276fb2 ({username='slave'} at 127.30.32.65:34313)
I20240502 20:46:17.663954 6723 tablet_copy_service.cc:161] P 7845c397230d49358ba2fad9db53f97c: Beginning new tablet copy session on tablet 6ff3c95ccfbf486fbc4bfc636afa829b from peer c92e6e46782849479c8a0c6d82276fb2 at {username='slave'} at 127.30.32.65:34313: session id = c92e6e46782849479c8a0c6d82276fb2-6ff3c95ccfbf486fbc4bfc636afa829b
I20240502 20:46:17.665503 6723 tablet_copy_source_session.cc:205] T 6ff3c95ccfbf486fbc4bfc636afa829b P 7845c397230d49358ba2fad9db53f97c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:17.666854 12000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ff3c95ccfbf486fbc4bfc636afa829b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:17.673894 12010 ts_tablet_manager.cc:902] T 253d158b340a444182793f18239bfa22 P 201d2b6485084e7dad2cbb722af94779: Initiating tablet copy from peer 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
F20240502 20:46:17.674096 12010 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-20373b53281a4214b27f014396f7a651.queue_time_us, tablet-download-pool-20373b53281a4214b27f014396f7a651.queue_time_us) (tablet-download-pool-20373b53281a4214b27f014396f7a651.run_cpu_time_us, tablet-download-pool-20373b53281a4214b27f014396f7a651.run_cpu_time_us) (tablet-download-pool-20373b53281a4214b27f014396f7a651.run_wall_time_us, tablet-download-pool-20373b53281a4214b27f014396f7a651.run_wall_time_us) (tablet-download-pool-253d158b340a444182793f18239bfa22.queue_time_us, tablet-download-pool-253d158b340a444182793f18239bfa22.queue_time_us) (tablet-download-pool-28029d4e66d1453cbde9811ffc2a3283.queue_time_us, tablet-download-pool-28029d4e66d1453cbde9811ffc2a3283.queue_time_us) (tablet-download-pool-28029d4e66d1453cbde9811ffc2a3283.run_cpu_time_us, tablet-download-pool-28029d4e66d1453cbde9811ffc2a3283.run_cpu_time_us) (tablet-download-pool-28029d4e66d1453cbde9811ffc2a3283.run_wall_time_us, tablet-download-pool-28029d4e66d1453cbde9811ffc2a3283.run_wall_time_us) (tablet-download-pool-41f84d5db2034af591b56e109be67934.queue_time_us, tablet-download-pool-41f84d5db2034af591b56e109be67934.queue_time_us) (tablet-download-pool-41f84d5db2034af591b56e109be67934.run_cpu_time_us, tablet-download-pool-41f84d5db2034af591b56e109be67934.run_cpu_time_us) (tablet-download-pool-41f84d5db2034af591b56e109be67934.run_wall_time_us, tablet-download-pool-41f84d5db2034af591b56e109be67934.run_wall_time_us) (tablet-download-pool-5a36bf6e8e6e43b2b47b6d6357b0d445.queue_time_us, tablet-download-pool-5a36bf6e8e6e43b2b47b6d6357b0d445.queue_time_us) (tablet-download-pool-5a36bf6e8e6e43b2b47b6d6357b0d445.run_cpu_time_us, tablet-download-pool-5a36bf6e8e6e43b2b47b6d6357b0d445.run_cpu_time_us) (tablet-download-pool-5a36bf6e8e6e43b2b47b6d6357b0d445.run_wall_time_us, tablet-download-pool-5a36bf6e8e6e43b2b47b6d6357b0d445.run_wall_time_us) (tablet-download-pool-76b6dee18d224e879bf0d3b6bbb2b9b5.queue_time_us, tablet-download-pool-76b6dee18d224e879bf0d3b6bbb2b9b5.queue_time_us) (tablet-download-pool-76b6dee18d224e879bf0d3b6bbb2b9b5.run_cpu_time_us, tablet-download-pool-76b6dee18d224e879bf0d3b6bbb2b9b5.run_cpu_time_us) (tablet-download-pool-76b6dee18d224e879bf0d3b6bbb2b9b5.run_wall_time_us, tablet-download-pool-76b6dee18d224e879bf0d3b6bbb2b9b5.run_wall_time_us) (tablet-download-pool-8e7e145762bf458a8374bfc503025a17.queue_time_us, tablet-download-pool-8e7e145762bf458a8374bfc503025a17.queue_time_us) (tablet-download-pool-8e7e145762bf458a8374bfc503025a17.run_cpu_time_us, tablet-download-pool-8e7e145762bf458a8374bfc503025a17.run_cpu_time_us) (tablet-download-pool-8e7e145762bf458a8374bfc503025a17.run_wall_time_us, tablet-download-pool-8e7e145762bf458a8374bfc503025a17.run_wall_time_us) (tablet-download-pool-9e34e8e17a7e413b97eb614a7df727d5.queue_time_us, tablet-download-pool-9e34e8e17a7e413b97eb614a7df727d5.queue_time_us) (tablet-download-pool-9e34e8e17a7e413b97eb614a7df727d5.run_cpu_time_us, tablet-download-pool-9e34e8e17a7e413b97eb614a7df727d5.run_cpu_time_us) (tablet-download-pool-9e34e8e17a7e413b97eb614a7df727d5.run_wall_time_us, tablet-download-pool-9e34e8e17a7e413b97eb614a7df727d5.run_wall_time_us) (tablet-download-pool-a7ea68e8581a4cd5a6212abc96dfbcde.queue_time_us, tablet-download-pool-a7ea68e8581a4cd5a6212abc96dfbcde.queue_time_us) (tablet-download-pool-a7ea68e8581a4cd5a6212abc96dfbcde.run_cpu_time_us, tablet-download-pool-a7ea68e8581a4cd5a6212abc96dfbcde.run_cpu_time_us) (tablet-download-pool-a7ea68e8581a4cd5a6212abc96dfbcde.run_wall_time_us, tablet-download-pool-a7ea68e8581a4cd5a6212abc96dfbcde.run_wall_time_us) (tablet-download-pool-cdf218d5e45b4db19a8e2c22dbc5cc5d.queue_time_us, tablet-download-pool-cdf218d5e45b4db19a8e2c22dbc5cc5d.queue_time_us) (tablet-download-pool-cdf218d5e45b4db19a8e2c22dbc5cc5d.run_cpu_time_us, tablet-download-pool-cdf218d5e45b4db19a8e2c22dbc5cc5d.run_cpu_time_us) (tablet-download-pool-cdf218d5e45b4db19a8e2c22dbc5cc5d.run_wall_time_us, tablet-download-pool-cdf218d5e45b4db19a8e2c22dbc5cc5d.run_wall_time_us) (tablet-download-pool-d37730b109d04b6395f75c7552524877.queue_time_us, tablet-download-pool-d37730b109d04b6395f75c7552524877.queue_time_us) (tablet-download-pool-d37730b109d04b6395f75c7552524877.run_cpu_time_us, tablet-download-pool-d37730b109d04b6395f75c7552524877.run_cpu_time_us) (tablet-download-pool-d37730b109d04b6395f75c7552524877.run_wall_time_us, tablet-download-pool-d37730b109d04b6395f75c7552524877.run_wall_time_us) (tablet-download-pool-d6fb394fdcff436886fe0634d79c024c.queue_time_us, tablet-download-pool-d6fb394fdcff436886fe0634d79c024c.queue_time_us) (tablet-download-pool-d6fb394fdcff436886fe0634d79c024c.run_cpu_time_us, tablet-download-pool-d6fb394fdcff436886fe0634d79c024c.run_cpu_time_us) (tablet-download-pool-d6fb394fdcff436886fe0634d79c024c.run_wall_time_us, tablet-download-pool-d6fb394fdcff436886fe0634d79c024c.run_wall_time_us) (tablet-download-pool-e5a4dc24914c4f37a14481b77c4fcdaa.queue_time_us, tablet-download-pool-e5a4dc24914c4f37a14481b77c4fcdaa.queue_time_us) (tablet-download-pool-e5a4dc24914c4f37a14481b77c4fcdaa.run_cpu_time_us, tablet-download-pool-e5a4dc24914c4f37a14481b77c4fcdaa.run_cpu_time_us) (tablet-download-pool-e5a4dc24914c4f37a14481b77c4fcdaa.run_wall_time_us, tablet-download-pool-e5a4dc24914c4f37a14481b77c4fcdaa.run_wall_time_us) (tablet-download-pool-eebddad3efe748f78372901ecad5b407.queue_time_us, tablet-download-pool-eebddad3efe748f78372901ecad5b407.queue_time_us) (tablet-download-pool-eebddad3efe748f78372901ecad5b407.run_cpu_time_us, tablet-download-pool-eebddad3efe748f78372901ecad5b407.run_cpu_time_us) (tablet-download-pool-eebddad3efe748f78372901ecad5b407.run_wall_time_us, tablet-download-pool-eebddad3efe748f78372901ecad5b407.run_wall_time_us) (tablet-download-pool-f352efa74d3c4da3907efdc37237211f.queue_time_us, tablet-download-pool-f352efa74d3c4da3907efdc37237211f.queue_time_us) (tablet-download-pool-f352efa74d3c4da3907efdc37237211f.run_cpu_time_us, tablet-download-pool-f352efa74d3c4da3907efdc37237211f.run_cpu_time_us) (tablet-download-pool-f352efa74d3c4da3907efdc37237211f.run_wall_time_us, tablet-download-pool-f352efa74d3c4da3907efdc37237211f.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f277f134dcd google::LogMessage::Fail() at ??:0
@ 0x7f277f138b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f277f1347cc google::LogMessage::Flush() at ??:0
@ 0x7f277f135f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240502 20:46:17.684711 12000 tablet_copy_client.cc:783] T 6ff3c95ccfbf486fbc4bfc636afa829b P c92e6e46782849479c8a0c6d82276fb2: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:17.684931 12000 tablet_copy_client.cc:647] T 6ff3c95ccfbf486fbc4bfc636afa829b P c92e6e46782849479c8a0c6d82276fb2: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:17.686364 12000 tablet_copy_client.cc:523] T 6ff3c95ccfbf486fbc4bfc636afa829b P c92e6e46782849479c8a0c6d82276fb2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
@ 0x7f278059e044 kudu::TraceMetrics::InternName() at ??:0
I20240502 20:46:17.688198 12000 tablet_bootstrap.cc:492] T 6ff3c95ccfbf486fbc4bfc636afa829b P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
@ 0x7f2780582f98 kudu::ThreadPool::ThreadPool() at ??:0
I20240502 20:46:17.692678 12013 ts_tablet_manager.cc:902] T 2c67f965350a45f89511679d40f04cbb P c3d26eafd51547acb780d4aac5ebe7cc: Initiating tablet copy from peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
I20240502 20:46:17.693751 12013 tablet_copy_client.cc:308] T 2c67f965350a45f89511679d40f04cbb P c3d26eafd51547acb780d4aac5ebe7cc: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.72:44643
I20240502 20:46:17.694191 12018 ts_tablet_manager.cc:902] T b69500b0144a42f59383b8c5d38324b3 P ed47c023452a497192f35d2f617bf9bd: Initiating tablet copy from peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
I20240502 20:46:17.694185 7119 tablet_copy_service.cc:140] P 201d2b6485084e7dad2cbb722af94779: Received BeginTabletCopySession request for tablet 2c67f965350a45f89511679d40f04cbb from peer c3d26eafd51547acb780d4aac5ebe7cc ({username='slave'} at 127.30.32.66:55487)
I20240502 20:46:17.694312 7119 tablet_copy_service.cc:161] P 201d2b6485084e7dad2cbb722af94779: Beginning new tablet copy session on tablet 2c67f965350a45f89511679d40f04cbb from peer c3d26eafd51547acb780d4aac5ebe7cc at {username='slave'} at 127.30.32.66:55487: session id = c3d26eafd51547acb780d4aac5ebe7cc-2c67f965350a45f89511679d40f04cbb
I20240502 20:46:17.694635 12018 tablet_copy_client.cc:308] T b69500b0144a42f59383b8c5d38324b3 P ed47c023452a497192f35d2f617bf9bd: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.72:44643
I20240502 20:46:17.695307 7118 tablet_copy_service.cc:140] P 201d2b6485084e7dad2cbb722af94779: Received BeginTabletCopySession request for tablet b69500b0144a42f59383b8c5d38324b3 from peer ed47c023452a497192f35d2f617bf9bd ({username='slave'} at 127.30.32.68:49707)
I20240502 20:46:17.695415 7119 tablet_copy_source_session.cc:205] T 2c67f965350a45f89511679d40f04cbb P 201d2b6485084e7dad2cbb722af94779: Tablet Copy: opened 0 blocks and 1 log segments
@ 0x7f2780580efc kudu::ThreadPoolBuilder::Build() at ??:0
I20240502 20:46:17.695425 7118 tablet_copy_service.cc:161] P 201d2b6485084e7dad2cbb722af94779: Beginning new tablet copy session on tablet b69500b0144a42f59383b8c5d38324b3 from peer ed47c023452a497192f35d2f617bf9bd at {username='slave'} at 127.30.32.68:49707: session id = ed47c023452a497192f35d2f617bf9bd-b69500b0144a42f59383b8c5d38324b3
I20240502 20:46:17.698415 12013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c67f965350a45f89511679d40f04cbb. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f278313377a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240502 20:46:17.696955 7118 tablet_copy_source_session.cc:205] T b69500b0144a42f59383b8c5d38324b3 P 201d2b6485084e7dad2cbb722af94779: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:17.703822 12018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b69500b0144a42f59383b8c5d38324b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:17.704322 12013 tablet_copy_client.cc:783] T 2c67f965350a45f89511679d40f04cbb P c3d26eafd51547acb780d4aac5ebe7cc: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:17.704502 12013 tablet_copy_client.cc:647] T 2c67f965350a45f89511679d40f04cbb P c3d26eafd51547acb780d4aac5ebe7cc: tablet copy: Starting download of 1 WAL segments...
@ 0x7f278313ca45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240502 20:46:17.707510 12018 tablet_copy_client.cc:783] T b69500b0144a42f59383b8c5d38324b3 P ed47c023452a497192f35d2f617bf9bd: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:17.707677 12018 tablet_copy_client.cc:647] T b69500b0144a42f59383b8c5d38324b3 P ed47c023452a497192f35d2f617bf9bd: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:17.709977 12018 tablet_copy_client.cc:523] T b69500b0144a42f59383b8c5d38324b3 P ed47c023452a497192f35d2f617bf9bd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
@ 0x7f27831bd76f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240502 20:46:17.713794 12013 tablet_copy_client.cc:523] T 2c67f965350a45f89511679d40f04cbb P c3d26eafd51547acb780d4aac5ebe7cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:17.714502 12018 tablet_bootstrap.cc:492] T b69500b0144a42f59383b8c5d38324b3 P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:46:17.715571 12013 tablet_bootstrap.cc:492] T 2c67f965350a45f89511679d40f04cbb P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:46:17.717190 12028 ts_tablet_manager.cc:902] T d78c02773e454ea0b79cf1282eda125b P 7845c397230d49358ba2fad9db53f97c: Initiating tablet copy from peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
@ 0x7f27831d14f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240502 20:46:17.717717 12028 tablet_copy_client.cc:308] T d78c02773e454ea0b79cf1282eda125b P 7845c397230d49358ba2fad9db53f97c: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.72:44643
I20240502 20:46:17.718195 7118 tablet_copy_service.cc:140] P 201d2b6485084e7dad2cbb722af94779: Received BeginTabletCopySession request for tablet d78c02773e454ea0b79cf1282eda125b from peer 7845c397230d49358ba2fad9db53f97c ({username='slave'} at 127.30.32.69:38849)
I20240502 20:46:17.718297 7118 tablet_copy_service.cc:161] P 201d2b6485084e7dad2cbb722af94779: Beginning new tablet copy session on tablet d78c02773e454ea0b79cf1282eda125b from peer 7845c397230d49358ba2fad9db53f97c at {username='slave'} at 127.30.32.69:38849: session id = 7845c397230d49358ba2fad9db53f97c-d78c02773e454ea0b79cf1282eda125b
I20240502 20:46:17.719306 7118 tablet_copy_source_session.cc:205] T d78c02773e454ea0b79cf1282eda125b P 201d2b6485084e7dad2cbb722af94779: Tablet Copy: opened 0 blocks and 1 log segments
@ 0x7f27831bc226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
I20240502 20:46:17.722441 12029 ts_tablet_manager.cc:902] T 3d938d01d5154590a9262acd54094ad2 P ed47c023452a497192f35d2f617bf9bd: Initiating tablet copy from peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
I20240502 20:46:17.722939 12029 tablet_copy_client.cc:308] T 3d938d01d5154590a9262acd54094ad2 P ed47c023452a497192f35d2f617bf9bd: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.69:42429
I20240502 20:46:17.723397 6723 tablet_copy_service.cc:140] P 7845c397230d49358ba2fad9db53f97c: Received BeginTabletCopySession request for tablet 3d938d01d5154590a9262acd54094ad2 from peer ed47c023452a497192f35d2f617bf9bd ({username='slave'} at 127.30.32.68:47929)
I20240502 20:46:17.723495 6723 tablet_copy_service.cc:161] P 7845c397230d49358ba2fad9db53f97c: Beginning new tablet copy session on tablet 3d938d01d5154590a9262acd54094ad2 from peer ed47c023452a497192f35d2f617bf9bd at {username='slave'} at 127.30.32.68:47929: session id = ed47c023452a497192f35d2f617bf9bd-3d938d01d5154590a9262acd54094ad2
I20240502 20:46:17.724494 6723 tablet_copy_source_session.cc:205] T 3d938d01d5154590a9262acd54094ad2 P 7845c397230d49358ba2fad9db53f97c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:17.725347 12028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d78c02773e454ea0b79cf1282eda125b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:17.725610 12029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d938d01d5154590a9262acd54094ad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:17.726240 12013 tablet_bootstrap.cc:492] T 2c67f965350a45f89511679d40f04cbb P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:17.726493 12013 tablet_bootstrap.cc:492] T 2c67f965350a45f89511679d40f04cbb P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap complete.
I20240502 20:46:17.727310 12013 ts_tablet_manager.cc:1367] T 2c67f965350a45f89511679d40f04cbb P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.006s
I20240502 20:46:17.728005 12028 tablet_copy_client.cc:783] T d78c02773e454ea0b79cf1282eda125b P 7845c397230d49358ba2fad9db53f97c: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:17.728138 12028 tablet_copy_client.cc:647] T d78c02773e454ea0b79cf1282eda125b P 7845c397230d49358ba2fad9db53f97c: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:17.728616 12029 tablet_copy_client.cc:783] T 3d938d01d5154590a9262acd54094ad2 P ed47c023452a497192f35d2f617bf9bd: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:17.728757 12029 tablet_copy_client.cc:647] T 3d938d01d5154590a9262acd54094ad2 P ed47c023452a497192f35d2f617bf9bd: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:17.728781 12013 raft_consensus.cc:348] T 2c67f965350a45f89511679d40f04cbb P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } }
I20240502 20:46:17.728919 12013 raft_consensus.cc:729] T 2c67f965350a45f89511679d40f04cbb P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: LEARNER
I20240502 20:46:17.729061 12013 consensus_queue.cc:260] T 2c67f965350a45f89511679d40f04cbb P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } }
I20240502 20:46:17.729784 12013 ts_tablet_manager.cc:1398] T 2c67f965350a45f89511679d40f04cbb P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:17.729257 12028 tablet_copy_client.cc:523] T d78c02773e454ea0b79cf1282eda125b P 7845c397230d49358ba2fad9db53f97c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:17.730273 7118 tablet_copy_service.cc:342] P 201d2b6485084e7dad2cbb722af94779: Request end of tablet copy session c3d26eafd51547acb780d4aac5ebe7cc-2c67f965350a45f89511679d40f04cbb received from {username='slave'} at 127.30.32.66:55487
I20240502 20:46:17.730345 7118 tablet_copy_service.cc:433] P 201d2b6485084e7dad2cbb722af94779: ending tablet copy session c3d26eafd51547acb780d4aac5ebe7cc-2c67f965350a45f89511679d40f04cbb on tablet 2c67f965350a45f89511679d40f04cbb with peer c3d26eafd51547acb780d4aac5ebe7cc
@ 0x7f27831cbc0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20240502 20:46:17.732347 12028 tablet_bootstrap.cc:492] T d78c02773e454ea0b79cf1282eda125b P 7845c397230d49358ba2fad9db53f97c: Bootstrap starting.
@ 0x55f36116675c std::function<>::operator()() at ??:0
I20240502 20:46:17.734875 12000 tablet_bootstrap.cc:492] T 6ff3c95ccfbf486fbc4bfc636afa829b P c92e6e46782849479c8a0c6d82276fb2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:17.735159 12000 tablet_bootstrap.cc:492] T 6ff3c95ccfbf486fbc4bfc636afa829b P c92e6e46782849479c8a0c6d82276fb2: Bootstrap complete.
I20240502 20:46:17.735350 12000 ts_tablet_manager.cc:1367] T 6ff3c95ccfbf486fbc4bfc636afa829b P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.047s user 0.000s sys 0.003s
I20240502 20:46:17.735957 12000 raft_consensus.cc:348] T 6ff3c95ccfbf486fbc4bfc636afa829b P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true } }
I20240502 20:46:17.736109 12000 raft_consensus.cc:729] T 6ff3c95ccfbf486fbc4bfc636afa829b P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: LEARNER
I20240502 20:46:17.736223 12000 consensus_queue.cc:260] T 6ff3c95ccfbf486fbc4bfc636afa829b P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true } }
@ 0x7f2780585e06 kudu::ThreadPool::DispatchThread() at ??:0
I20240502 20:46:17.736537 12000 ts_tablet_manager.cc:1398] T 6ff3c95ccfbf486fbc4bfc636afa829b P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.002s
I20240502 20:46:17.737026 12018 tablet_bootstrap.cc:492] T b69500b0144a42f59383b8c5d38324b3 P ed47c023452a497192f35d2f617bf9bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:17.737213 12029 tablet_copy_client.cc:523] T 3d938d01d5154590a9262acd54094ad2 P ed47c023452a497192f35d2f617bf9bd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:17.737308 12018 tablet_bootstrap.cc:492] T b69500b0144a42f59383b8c5d38324b3 P ed47c023452a497192f35d2f617bf9bd: Bootstrap complete.
I20240502 20:46:17.737463 12018 ts_tablet_manager.cc:1367] T b69500b0144a42f59383b8c5d38324b3 P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.004s
I20240502 20:46:17.737844 6723 tablet_copy_service.cc:342] P 7845c397230d49358ba2fad9db53f97c: Request end of tablet copy session c92e6e46782849479c8a0c6d82276fb2-6ff3c95ccfbf486fbc4bfc636afa829b received from {username='slave'} at 127.30.32.65:34313
I20240502 20:46:17.737917 6723 tablet_copy_service.cc:433] P 7845c397230d49358ba2fad9db53f97c: ending tablet copy session c92e6e46782849479c8a0c6d82276fb2-6ff3c95ccfbf486fbc4bfc636afa829b on tablet 6ff3c95ccfbf486fbc4bfc636afa829b with peer c92e6e46782849479c8a0c6d82276fb2
I20240502 20:46:17.737938 12018 raft_consensus.cc:348] T b69500b0144a42f59383b8c5d38324b3 P ed47c023452a497192f35d2f617bf9bd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: true } }
I20240502 20:46:17.738044 12018 raft_consensus.cc:729] T b69500b0144a42f59383b8c5d38324b3 P ed47c023452a497192f35d2f617bf9bd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: LEARNER
I20240502 20:46:17.738134 12018 consensus_queue.cc:260] T b69500b0144a42f59383b8c5d38324b3 P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: true } }
I20240502 20:46:17.738421 12018 ts_tablet_manager.cc:1398] T b69500b0144a42f59383b8c5d38324b3 P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:17.739240 4231 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41812:
name: "rebalancer_extra_table_28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 20:46:17.743314 4235 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 20:46:17.745079 12013 ts_tablet_manager.cc:902] T a999f994b78a4b33a95ee7d5407cba20 P c3d26eafd51547acb780d4aac5ebe7cc: Initiating tablet copy from peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
I20240502 20:46:17.745574 12013 tablet_copy_client.cc:308] T a999f994b78a4b33a95ee7d5407cba20 P c3d26eafd51547acb780d4aac5ebe7cc: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.71:46841
I20240502 20:46:17.747175 12029 tablet_bootstrap.cc:492] T 3d938d01d5154590a9262acd54094ad2 P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:46:17.748433 6987 tablet_copy_service.cc:140] P a47e5c56f13843018ac584914a8d4707: Received BeginTabletCopySession request for tablet a999f994b78a4b33a95ee7d5407cba20 from peer c3d26eafd51547acb780d4aac5ebe7cc ({username='slave'} at 127.30.32.66:49749)
I20240502 20:46:17.748528 6987 tablet_copy_service.cc:161] P a47e5c56f13843018ac584914a8d4707: Beginning new tablet copy session on tablet a999f994b78a4b33a95ee7d5407cba20 from peer c3d26eafd51547acb780d4aac5ebe7cc at {username='slave'} at 127.30.32.66:49749: session id = c3d26eafd51547acb780d4aac5ebe7cc-a999f994b78a4b33a95ee7d5407cba20
I20240502 20:46:17.749593 6987 tablet_copy_source_session.cc:205] T a999f994b78a4b33a95ee7d5407cba20 P a47e5c56f13843018ac584914a8d4707: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:17.749651 12028 tablet_bootstrap.cc:492] T d78c02773e454ea0b79cf1282eda125b P 7845c397230d49358ba2fad9db53f97c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:17.749897 12028 tablet_bootstrap.cc:492] T d78c02773e454ea0b79cf1282eda125b P 7845c397230d49358ba2fad9db53f97c: Bootstrap complete.
I20240502 20:46:17.750043 12028 ts_tablet_manager.cc:1367] T d78c02773e454ea0b79cf1282eda125b P 7845c397230d49358ba2fad9db53f97c: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20240502 20:46:17.750478 12028 raft_consensus.cc:348] T d78c02773e454ea0b79cf1282eda125b P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } }
I20240502 20:46:17.750605 12028 raft_consensus.cc:729] T d78c02773e454ea0b79cf1282eda125b P 7845c397230d49358ba2fad9db53f97c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7845c397230d49358ba2fad9db53f97c, State: Initialized, Role: LEARNER
I20240502 20:46:17.750695 12028 consensus_queue.cc:260] T d78c02773e454ea0b79cf1282eda125b P 7845c397230d49358ba2fad9db53f97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } }
I20240502 20:46:17.751274 12028 ts_tablet_manager.cc:1398] T d78c02773e454ea0b79cf1282eda125b P 7845c397230d49358ba2fad9db53f97c: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 20:46:17.752035 12013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a999f994b78a4b33a95ee7d5407cba20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:17.753036 7118 tablet_copy_service.cc:342] P 201d2b6485084e7dad2cbb722af94779: Request end of tablet copy session 7845c397230d49358ba2fad9db53f97c-d78c02773e454ea0b79cf1282eda125b received from {username='slave'} at 127.30.32.69:38849
I20240502 20:46:17.753119 7118 tablet_copy_service.cc:433] P 201d2b6485084e7dad2cbb722af94779: ending tablet copy session 7845c397230d49358ba2fad9db53f97c-d78c02773e454ea0b79cf1282eda125b on tablet d78c02773e454ea0b79cf1282eda125b with peer 7845c397230d49358ba2fad9db53f97c
I20240502 20:46:17.755647 12013 tablet_copy_client.cc:783] T a999f994b78a4b33a95ee7d5407cba20 P c3d26eafd51547acb780d4aac5ebe7cc: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:17.755812 12013 tablet_copy_client.cc:647] T a999f994b78a4b33a95ee7d5407cba20 P c3d26eafd51547acb780d4aac5ebe7cc: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:17.758059 7118 tablet_copy_service.cc:342] P 201d2b6485084e7dad2cbb722af94779: Request end of tablet copy session ed47c023452a497192f35d2f617bf9bd-b69500b0144a42f59383b8c5d38324b3 received from {username='slave'} at 127.30.32.68:49707
I20240502 20:46:17.758137 7118 tablet_copy_service.cc:433] P 201d2b6485084e7dad2cbb722af94779: ending tablet copy session ed47c023452a497192f35d2f617bf9bd-b69500b0144a42f59383b8c5d38324b3 on tablet b69500b0144a42f59383b8c5d38324b3 with peer ed47c023452a497192f35d2f617bf9bd
I20240502 20:46:17.760896 11983 tablet.cc:1722] T 2a44b2af7886427db821f873e5a79151 P a47e5c56f13843018ac584914a8d4707: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 20:46:17.761438 11984 tablet.cc:1722] T 2a44b2af7886427db821f873e5a79151 P 201d2b6485084e7dad2cbb722af94779: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 20:46:17.763595 11985 tablet.cc:1722] T 2a44b2af7886427db821f873e5a79151 P a4bf1df7fa6b4948be502c182bfbb833: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 20:46:17.767704 7079 tablet_service.cc:1467] Processing CreateTablet for tablet a572c30cd4584ec498a22c1ce30c7e3a (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=7c9bc5b12d25455e854a7e0d4e78a6a2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:17.767989 7079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a572c30cd4584ec498a22c1ce30c7e3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:17.768291 4487 tablet_service.cc:1467] Processing CreateTablet for tablet c60371165de94a5d9ec1c7181f839e5b (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=7c9bc5b12d25455e854a7e0d4e78a6a2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:17.768532 4487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c60371165de94a5d9ec1c7181f839e5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:17.768515 6681 tablet_service.cc:1467] Processing CreateTablet for tablet 7ca68870fae3480786c84bd86dc4897e (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=7c9bc5b12d25455e854a7e0d4e78a6a2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:17.768793 6681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ca68870fae3480786c84bd86dc4897e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:17.769243 6947 tablet_service.cc:1467] Processing CreateTablet for tablet a572c30cd4584ec498a22c1ce30c7e3a (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=7c9bc5b12d25455e854a7e0d4e78a6a2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:17.769481 6947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a572c30cd4584ec498a22c1ce30c7e3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:17.770453 7077 tablet_service.cc:1467] Processing CreateTablet for tablet c60371165de94a5d9ec1c7181f839e5b (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=7c9bc5b12d25455e854a7e0d4e78a6a2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:17.770684 7077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c60371165de94a5d9ec1c7181f839e5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:17.772338 12013 tablet_copy_client.cc:523] T a999f994b78a4b33a95ee7d5407cba20 P c3d26eafd51547acb780d4aac5ebe7cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:17.772799 4750 tablet_service.cc:1467] Processing CreateTablet for tablet 7ca68870fae3480786c84bd86dc4897e (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=7c9bc5b12d25455e854a7e0d4e78a6a2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:17.773264 6683 tablet_service.cc:1467] Processing CreateTablet for tablet a572c30cd4584ec498a22c1ce30c7e3a (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=7c9bc5b12d25455e854a7e0d4e78a6a2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:17.773553 6683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a572c30cd4584ec498a22c1ce30c7e3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:17.773756 12013 tablet_bootstrap.cc:492] T a999f994b78a4b33a95ee7d5407cba20 P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:46:17.777436 12000 ts_tablet_manager.cc:902] T 533e7f3c852c4fd9bd5093085a5e8e36 P c92e6e46782849479c8a0c6d82276fb2: Initiating tablet copy from peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
@ 0x7f27805866ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
I20240502 20:46:17.777889 12000 tablet_copy_client.cc:308] T 533e7f3c852c4fd9bd5093085a5e8e36 P c92e6e46782849479c8a0c6d82276fb2: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.69:42429
I20240502 20:46:17.778467 6723 tablet_copy_service.cc:140] P 7845c397230d49358ba2fad9db53f97c: Received BeginTabletCopySession request for tablet 533e7f3c852c4fd9bd5093085a5e8e36 from peer c92e6e46782849479c8a0c6d82276fb2 ({username='slave'} at 127.30.32.65:34313)
I20240502 20:46:17.778577 6723 tablet_copy_service.cc:161] P 7845c397230d49358ba2fad9db53f97c: Beginning new tablet copy session on tablet 533e7f3c852c4fd9bd5093085a5e8e36 from peer c92e6e46782849479c8a0c6d82276fb2 at {username='slave'} at 127.30.32.65:34313: session id = c92e6e46782849479c8a0c6d82276fb2-533e7f3c852c4fd9bd5093085a5e8e36
I20240502 20:46:17.778792 4750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ca68870fae3480786c84bd86dc4897e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:17.779709 6723 tablet_copy_source_session.cc:205] T 533e7f3c852c4fd9bd5093085a5e8e36 P 7845c397230d49358ba2fad9db53f97c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:17.780593 12000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 533e7f3c852c4fd9bd5093085a5e8e36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:17.780808 12013 tablet_bootstrap.cc:492] T a999f994b78a4b33a95ee7d5407cba20 P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:17.781234 6815 tablet_service.cc:1467] Processing CreateTablet for tablet c60371165de94a5d9ec1c7181f839e5b (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=7c9bc5b12d25455e854a7e0d4e78a6a2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:17.781493 6815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c60371165de94a5d9ec1c7181f839e5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:17.781527 12013 tablet_bootstrap.cc:492] T a999f994b78a4b33a95ee7d5407cba20 P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap complete.
I20240502 20:46:17.781683 12013 ts_tablet_manager.cc:1367] T a999f994b78a4b33a95ee7d5407cba20 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240502 20:46:17.782155 12013 raft_consensus.cc:348] T a999f994b78a4b33a95ee7d5407cba20 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } }
I20240502 20:46:17.782282 12013 raft_consensus.cc:729] T a999f994b78a4b33a95ee7d5407cba20 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: LEARNER
I20240502 20:46:17.782397 12013 consensus_queue.cc:260] T a999f994b78a4b33a95ee7d5407cba20 P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } }
I20240502 20:46:17.782799 12013 ts_tablet_manager.cc:1398] T a999f994b78a4b33a95ee7d5407cba20 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240502 20:46:17.782886 12000 tablet_copy_client.cc:783] T 533e7f3c852c4fd9bd5093085a5e8e36 P c92e6e46782849479c8a0c6d82276fb2: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:17.783018 12000 tablet_copy_client.cc:647] T 533e7f3c852c4fd9bd5093085a5e8e36 P c92e6e46782849479c8a0c6d82276fb2: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:17.783290 6802 tablet_service.cc:1467] Processing CreateTablet for tablet 7ca68870fae3480786c84bd86dc4897e (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=7c9bc5b12d25455e854a7e0d4e78a6a2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:17.783528 6802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ca68870fae3480786c84bd86dc4897e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:17.783888 6987 tablet_copy_service.cc:342] P a47e5c56f13843018ac584914a8d4707: Request end of tablet copy session c3d26eafd51547acb780d4aac5ebe7cc-a999f994b78a4b33a95ee7d5407cba20 received from {username='slave'} at 127.30.32.66:49749
I20240502 20:46:17.783968 6987 tablet_copy_service.cc:433] P a47e5c56f13843018ac584914a8d4707: ending tablet copy session c3d26eafd51547acb780d4aac5ebe7cc-a999f994b78a4b33a95ee7d5407cba20 on tablet a999f994b78a4b33a95ee7d5407cba20 with peer c3d26eafd51547acb780d4aac5ebe7cc
I20240502 20:46:17.784790 12028 ts_tablet_manager.cc:902] T 73efeba3d3094ef387e4efb03cff9d4e P 7845c397230d49358ba2fad9db53f97c: Initiating tablet copy from peer ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
F20240502 20:46:17.784948 12028 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-00eb91e624554baf918e96b098019be8.queue_time_us, tablet-download-pool-00eb91e624554baf918e96b098019be8.queue_time_us) (tablet-download-pool-00eb91e624554baf918e96b098019be8.run_cpu_time_us, tablet-download-pool-00eb91e624554baf918e96b098019be8.run_cpu_time_us) (tablet-download-pool-00eb91e624554baf918e96b098019be8.run_wall_time_us, tablet-download-pool-00eb91e624554baf918e96b098019be8.run_wall_time_us) (tablet-download-pool-02302fc79dc04b0396041c600c2f4bef.queue_time_us, tablet-download-pool-02302fc79dc04b0396041c600c2f4bef.queue_time_us) (tablet-download-pool-02302fc79dc04b0396041c600c2f4bef.run_cpu_time_us, tablet-download-pool-02302fc79dc04b0396041c600c2f4bef.run_cpu_time_us) (tablet-download-pool-02302fc79dc04b0396041c600c2f4bef.run_wall_time_us, tablet-download-pool-02302fc79dc04b0396041c600c2f4bef.run_wall_time_us) (tablet-download-pool-04c72d7d63574dea99cb2f7c14199a73.queue_time_us, tablet-download-pool-04c72d7d63574dea99cb2f7c14199a73.queue_time_us) (tablet-download-pool-04c72d7d63574dea99cb2f7c14199a73.run_cpu_time_us, tablet-download-pool-04c72d7d63574dea99cb2f7c14199a73.run_cpu_time_us) (tablet-download-pool-04c72d7d63574dea99cb2f7c14199a73.run_wall_time_us, tablet-download-pool-04c72d7d63574dea99cb2f7c14199a73.run_wall_time_us) (tablet-download-pool-180902058815460189f75de4704d1e84.queue_time_us, tablet-download-pool-180902058815460189f75de4704d1e84.queue_time_us) (tablet-download-pool-180902058815460189f75de4704d1e84.run_cpu_time_us, tablet-download-pool-180902058815460189f75de4704d1e84.run_cpu_time_us) (tablet-download-pool-180902058815460189f75de4704d1e84.run_wall_time_us, tablet-download-pool-180902058815460189f75de4704d1e84.run_wall_time_us) (tablet-download-pool-73efeba3d3094ef387e4efb03cff9d4e.queue_time_us, tablet-download-pool-73efeba3d3094ef387e4efb03cff9d4e.queue_time_us) (tablet-download-pool-8565ebbd21dd4f708a0254d1ed2d6670.queue_time_us, tablet-download-pool-8565ebbd21dd4f708a0254d1ed2d6670.queue_time_us) (tablet-download-pool-8565ebbd21dd4f708a0254d1ed2d6670.run_cpu_time_us, tablet-download-pool-8565ebbd21dd4f708a0254d1ed2d6670.run_cpu_time_us) (tablet-download-pool-8565ebbd21dd4f708a0254d1ed2d6670.run_wall_time_us, tablet-download-pool-8565ebbd21dd4f708a0254d1ed2d6670.run_wall_time_us) (tablet-download-pool-93b7f7de1f8c49389c3d8c1e03c67f26.queue_time_us, tablet-download-pool-93b7f7de1f8c49389c3d8c1e03c67f26.queue_time_us) (tablet-download-pool-93b7f7de1f8c49389c3d8c1e03c67f26.run_cpu_time_us, tablet-download-pool-93b7f7de1f8c49389c3d8c1e03c67f26.run_cpu_time_us) (tablet-download-pool-93b7f7de1f8c49389c3d8c1e03c67f26.run_wall_time_us, tablet-download-pool-93b7f7de1f8c49389c3d8c1e03c67f26.run_wall_time_us) (tablet-download-pool-a2e81d2c63cd4e4cac5086506b5c4a12.queue_time_us, tablet-download-pool-a2e81d2c63cd4e4cac5086506b5c4a12.queue_time_us) (tablet-download-pool-a2e81d2c63cd4e4cac5086506b5c4a12.run_cpu_time_us, tablet-download-pool-a2e81d2c63cd4e4cac5086506b5c4a12.run_cpu_time_us) (tablet-download-pool-a2e81d2c63cd4e4cac5086506b5c4a12.run_wall_time_us, tablet-download-pool-a2e81d2c63cd4e4cac5086506b5c4a12.run_wall_time_us) (tablet-download-pool-a55d5b8b9ed04ca39888b89b82a3f833.queue_time_us, tablet-download-pool-a55d5b8b9ed04ca39888b89b82a3f833.queue_time_us) (tablet-download-pool-a55d5b8b9ed04ca39888b89b82a3f833.run_cpu_time_us, tablet-download-pool-a55d5b8b9ed04ca39888b89b82a3f833.run_cpu_time_us) (tablet-download-pool-a55d5b8b9ed04ca39888b89b82a3f833.run_wall_time_us, tablet-download-pool-a55d5b8b9ed04ca39888b89b82a3f833.run_wall_time_us) (tablet-download-pool-a62a7ce526f14c25994bb9e0e3c3bc84.queue_time_us, tablet-download-pool-a62a7ce526f14c25994bb9e0e3c3bc84.queue_time_us) (tablet-download-pool-a62a7ce526f14c25994bb9e0e3c3bc84.run_cpu_time_us, tablet-download-pool-a62a7ce526f14c25994bb9e0e3c3bc84.run_cpu_time_us) (tablet-download-pool-a62a7ce526f14c25994bb9e0e3c3bc84.run_wall_time_us, tablet-download-pool-a62a7ce526f14c25994bb9e0e3c3bc84.run_wall_time_us) (tablet-download-pool-c9d25657e9a14b35b2f6e8d82f4533c7.queue_time_us, tablet-download-pool-c9d25657e9a14b35b2f6e8d82f4533c7.queue_time_us) (tablet-download-pool-c9d25657e9a14b35b2f6e8d82f4533c7.run_cpu_time_us, tablet-download-pool-c9d25657e9a14b35b2f6e8d82f4533c7.run_cpu_time_us) (tablet-download-pool-c9d25657e9a14b35b2f6e8d82f4533c7.run_wall_time_us, tablet-download-pool-c9d25657e9a14b35b2f6e8d82f4533c7.run_wall_time_us) (tablet-download-pool-d37fab152a8a44868d3160ec3057bb98.queue_time_us, tablet-download-pool-d37fab152a8a44868d3160ec3057bb98.queue_time_us) (tablet-download-pool-d37fab152a8a44868d3160ec3057bb98.run_cpu_time_us, tablet-download-pool-d37fab152a8a44868d3160ec3057bb98.run_cpu_time_us) (tablet-download-pool-d37fab152a8a44868d3160ec3057bb98.run_wall_time_us, tablet-download-pool-d37fab152a8a44868d3160ec3057bb98.run_wall_time_us) (tablet-download-pool-d6582234196c4fe4a82b3182787700de.queue_time_us, tablet-download-pool-d6582234196c4fe4a82b3182787700de.queue_time_us) (tablet-download-pool-d6582234196c4fe4a82b3182787700de.run_cpu_time_us, tablet-download-pool-d6582234196c4fe4a82b3182787700de.run_cpu_time_us) (tablet-download-pool-d6582234196c4fe4a82b3182787700de.run_wall_time_us, tablet-download-pool-d6582234196c4fe4a82b3182787700de.run_wall_time_us) (tablet-download-pool-d6fb394fdcff436886fe0634d79c024c.queue_time_us, tablet-download-pool-d6fb394fdcff436886fe0634d79c024c.queue_time_us) (tablet-download-pool-d6fb394fdcff436886fe0634d79c024c.run_cpu_time_us, tablet-download-pool-d6fb394fdcff436886fe0634d79c024c.run_cpu_time_us) (tablet-download-pool-d6fb394fdcff436886fe0634d79c024c.run_wall_time_us, tablet-download-pool-d6fb394fdcff436886fe0634d79c024c.run_wall_time_us) (tablet-download-pool-d78c02773e454ea0b79cf1282eda125b.queue_time_us, tablet-download-pool-d78c02773e454ea0b79cf1282eda125b.queue_time_us) (tablet-download-pool-d78c02773e454ea0b79cf1282eda125b.run_cpu_time_us, tablet-download-pool-d78c02773e454ea0b79cf1282eda125b.run_cpu_time_us) (tablet-download-pool-d78c02773e454ea0b79cf1282eda125b.run_wall_time_us, tablet-download-pool-d78c02773e454ea0b79cf1282eda125b.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240502 20:46:17.787770 4233 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: rebalancer_dynamic_table_new_name [id=c863a144af7347ea87a5e829dd2608be] alter complete (version 3)
I20240502 20:46:17.787834 12029 tablet_bootstrap.cc:492] T 3d938d01d5154590a9262acd54094ad2 P ed47c023452a497192f35d2f617bf9bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:17.788118 12029 tablet_bootstrap.cc:492] T 3d938d01d5154590a9262acd54094ad2 P ed47c023452a497192f35d2f617bf9bd: Bootstrap complete.
I20240502 20:46:17.789602 12055 tablet_bootstrap.cc:492] T c60371165de94a5d9ec1c7181f839e5b P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
@ 0x7f4ff54b2dcd google::LogMessage::Fail() at ??:0
I20240502 20:46:17.790205 12029 ts_tablet_manager.cc:1367] T 3d938d01d5154590a9262acd54094ad2 P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.043s user 0.007s sys 0.000s
@ 0x7f278058803b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20240502 20:46:17.791875 12056 tablet_bootstrap.cc:492] T a572c30cd4584ec498a22c1ce30c7e3a P a47e5c56f13843018ac584914a8d4707: Bootstrap starting.
@ 0x7f4ff54b6b93 google::LogMessage::SendToLog() at ??:0
I20240502 20:46:17.792620 12029 raft_consensus.cc:348] T 3d938d01d5154590a9262acd54094ad2 P ed47c023452a497192f35d2f617bf9bd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: true } }
I20240502 20:46:17.792835 12029 raft_consensus.cc:729] T 3d938d01d5154590a9262acd54094ad2 P ed47c023452a497192f35d2f617bf9bd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: LEARNER
I20240502 20:46:17.792954 12029 consensus_queue.cc:260] T 3d938d01d5154590a9262acd54094ad2 P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: true } }
I20240502 20:46:17.794587 12029 ts_tablet_manager.cc:1398] T 3d938d01d5154590a9262acd54094ad2 P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
@ 0x7f4ff54b27cc google::LogMessage::Flush() at ??:0
I20240502 20:46:17.795063 6723 tablet_copy_service.cc:342] P 7845c397230d49358ba2fad9db53f97c: Request end of tablet copy session ed47c023452a497192f35d2f617bf9bd-3d938d01d5154590a9262acd54094ad2 received from {username='slave'} at 127.30.32.68:47929
I20240502 20:46:17.795141 6723 tablet_copy_service.cc:433] P 7845c397230d49358ba2fad9db53f97c: ending tablet copy session ed47c023452a497192f35d2f617bf9bd-3d938d01d5154590a9262acd54094ad2 on tablet 3d938d01d5154590a9262acd54094ad2 with peer ed47c023452a497192f35d2f617bf9bd
@ 0x55f36116675c std::function<>::operator()() at ??:0
I20240502 20:46:17.798076 12057 tablet_bootstrap.cc:492] T 7ca68870fae3480786c84bd86dc4897e P 7845c397230d49358ba2fad9db53f97c: Bootstrap starting.
I20240502 20:46:17.798418 12055 tablet_bootstrap.cc:654] T c60371165de94a5d9ec1c7181f839e5b P c3d26eafd51547acb780d4aac5ebe7cc: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:17.799908 12065 ts_tablet_manager.cc:902] T ef4002126e8847809add182f262d62d9 P c92e6e46782849479c8a0c6d82276fb2: Initiating tablet copy from peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
@ 0x7f4ff54b3f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240502 20:46:17.800962 12057 tablet_bootstrap.cc:654] T 7ca68870fae3480786c84bd86dc4897e P 7845c397230d49358ba2fad9db53f97c: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:17.801733 12000 tablet_copy_client.cc:523] T 533e7f3c852c4fd9bd5093085a5e8e36 P c92e6e46782849479c8a0c6d82276fb2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
@ 0x7f2780575807 kudu::Thread::SuperviseThread() at ??:0
I20240502 20:46:17.803682 12000 tablet_bootstrap.cc:492] T 533e7f3c852c4fd9bd5093085a5e8e36 P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:46:17.805191 12056 tablet_bootstrap.cc:654] T a572c30cd4584ec498a22c1ce30c7e3a P a47e5c56f13843018ac584914a8d4707: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:17.805200 12063 tablet_bootstrap.cc:492] T 7ca68870fae3480786c84bd86dc4897e P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
@ 0x7f2780db36db start_thread at ??:0
I20240502 20:46:17.806187 12063 tablet_bootstrap.cc:654] T 7ca68870fae3480786c84bd86dc4897e P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:17.807106 12065 tablet_copy_client.cc:308] T ef4002126e8847809add182f262d62d9 P c92e6e46782849479c8a0c6d82276fb2: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.71:46841
I20240502 20:46:17.807133 12056 tablet_bootstrap.cc:492] T a572c30cd4584ec498a22c1ce30c7e3a P a47e5c56f13843018ac584914a8d4707: No bootstrap required, opened a new log
I20240502 20:46:17.807209 12064 tablet_bootstrap.cc:492] T c60371165de94a5d9ec1c7181f839e5b P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap starting.
I20240502 20:46:17.807228 12056 ts_tablet_manager.cc:1367] T a572c30cd4584ec498a22c1ce30c7e3a P a47e5c56f13843018ac584914a8d4707: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240502 20:46:17.807559 6987 tablet_copy_service.cc:140] P a47e5c56f13843018ac584914a8d4707: Received BeginTabletCopySession request for tablet ef4002126e8847809add182f262d62d9 from peer c92e6e46782849479c8a0c6d82276fb2 ({username='slave'} at 127.30.32.65:42487)
I20240502 20:46:17.807636 6987 tablet_copy_service.cc:161] P a47e5c56f13843018ac584914a8d4707: Beginning new tablet copy session on tablet ef4002126e8847809add182f262d62d9 from peer c92e6e46782849479c8a0c6d82276fb2 at {username='slave'} at 127.30.32.65:42487: session id = c92e6e46782849479c8a0c6d82276fb2-ef4002126e8847809add182f262d62d9
I20240502 20:46:17.807780 12056 raft_consensus.cc:348] T a572c30cd4584ec498a22c1ce30c7e3a P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:17.807886 12056 raft_consensus.cc:374] T a572c30cd4584ec498a22c1ce30c7e3a P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:17.807965 12056 raft_consensus.cc:729] T a572c30cd4584ec498a22c1ce30c7e3a P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47e5c56f13843018ac584914a8d4707, State: Initialized, Role: FOLLOWER
I20240502 20:46:17.808077 12056 consensus_queue.cc:260] T a572c30cd4584ec498a22c1ce30c7e3a P a47e5c56f13843018ac584914a8d4707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:17.808418 12064 tablet_bootstrap.cc:654] T c60371165de94a5d9ec1c7181f839e5b P a4bf1df7fa6b4948be502c182bfbb833: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:17.808650 6987 tablet_copy_source_session.cc:205] T ef4002126e8847809add182f262d62d9 P a47e5c56f13843018ac584914a8d4707: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:17.810308 10138 raft_consensus.cc:482] T a572c30cd4584ec498a22c1ce30c7e3a P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:17.810391 10138 raft_consensus.cc:504] T a572c30cd4584ec498a22c1ce30c7e3a P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:17.810810 10138 leader_election.cc:288] T a572c30cd4584ec498a22c1ce30c7e3a P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
I20240502 20:46:17.811100 12065 ts_tablet_manager.cc:1875] T ef4002126e8847809add182f262d62d9 P c92e6e46782849479c8a0c6d82276fb2: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 20:46:17.811189 6702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a572c30cd4584ec498a22c1ce30c7e3a" candidate_uuid: "a47e5c56f13843018ac584914a8d4707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7845c397230d49358ba2fad9db53f97c" is_pre_election: true
I20240502 20:46:17.811321 7098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a572c30cd4584ec498a22c1ce30c7e3a" candidate_uuid: "a47e5c56f13843018ac584914a8d4707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201d2b6485084e7dad2cbb722af94779" is_pre_election: true
W20240502 20:46:17.811570 6902 leader_election.cc:341] T a572c30cd4584ec498a22c1ce30c7e3a P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Illegal state: must be running to vote when last-logged opid is not known
W20240502 20:46:17.811691 6900 leader_election.cc:341] T a572c30cd4584ec498a22c1ce30c7e3a P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Illegal state: must be running to vote when last-logged opid is not known
@ 0x7f277e2eb71f clone at ??:0
I20240502 20:46:17.811794 6900 leader_election.cc:302] T a572c30cd4584ec498a22c1ce30c7e3a P a47e5c56f13843018ac584914a8d4707 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a47e5c56f13843018ac584914a8d4707; no voters: 201d2b6485084e7dad2cbb722af94779, 7845c397230d49358ba2fad9db53f97c
I20240502 20:46:17.812343 10138 raft_consensus.cc:2726] T a572c30cd4584ec498a22c1ce30c7e3a P a47e5c56f13843018ac584914a8d4707 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 20:46:17.813160 12056 ts_tablet_manager.cc:1398] T a572c30cd4584ec498a22c1ce30c7e3a P a47e5c56f13843018ac584914a8d4707: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 20:46:17.813675 12080 ts_tablet_manager.cc:902] T cedd1c79256a4f32a73b6a88244e6181 P a47e5c56f13843018ac584914a8d4707: Initiating tablet copy from peer c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827)
F20240502 20:46:17.813838 12080 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-00eb91e624554baf918e96b098019be8.queue_time_us, tablet-download-pool-00eb91e624554baf918e96b098019be8.queue_time_us) (tablet-download-pool-00eb91e624554baf918e96b098019be8.run_cpu_time_us, tablet-download-pool-00eb91e624554baf918e96b098019be8.run_cpu_time_us) (tablet-download-pool-00eb91e624554baf918e96b098019be8.run_wall_time_us, tablet-download-pool-00eb91e624554baf918e96b098019be8.run_wall_time_us) (tablet-download-pool-0cf7e95cb91b4e74b14365fcc6b7f968.queue_time_us, tablet-download-pool-0cf7e95cb91b4e74b14365fcc6b7f968.queue_time_us) (tablet-download-pool-0cf7e95cb91b4e74b14365fcc6b7f968.run_cpu_time_us, tablet-download-pool-0cf7e95cb91b4e74b14365fcc6b7f968.run_cpu_time_us) (tablet-download-pool-0cf7e95cb91b4e74b14365fcc6b7f968.run_wall_time_us, tablet-download-pool-0cf7e95cb91b4e74b14365fcc6b7f968.run_wall_time_us) (tablet-download-pool-28da7347f982468281570ffa7503a854.queue_time_us, tablet-download-pool-28da7347f982468281570ffa7503a854.queue_time_us) (tablet-download-pool-28da7347f982468281570ffa7503a854.run_cpu_time_us, tablet-download-pool-28da7347f982468281570ffa7503a854.run_cpu_time_us) (tablet-download-pool-28da7347f982468281570ffa7503a854.run_wall_time_us, tablet-download-pool-28da7347f982468281570ffa7503a854.run_wall_time_us) (tablet-download-pool-428ffbac75db466191cb3bef63b4eb79.queue_time_us, tablet-download-pool-428ffbac75db466191cb3bef63b4eb79.queue_time_us) (tablet-download-pool-428ffbac75db466191cb3bef63b4eb79.run_cpu_time_us, tablet-download-pool-428ffbac75db466191cb3bef63b4eb79.run_cpu_time_us) (tablet-download-pool-428ffbac75db466191cb3bef63b4eb79.run_wall_time_us, tablet-download-pool-428ffbac75db466191cb3bef63b4eb79.run_wall_time_us) (tablet-download-pool-45ade8c672d8445eaffb179772e3759a.queue_time_us, tablet-download-pool-45ade8c672d8445eaffb179772e3759a.queue_time_us) (tablet-download-pool-45ade8c672d8445eaffb179772e3759a.run_cpu_time_us, tablet-download-pool-45ade8c672d8445eaffb179772e3759a.run_cpu_time_us) (tablet-download-pool-45ade8c672d8445eaffb179772e3759a.run_wall_time_us, tablet-download-pool-45ade8c672d8445eaffb179772e3759a.run_wall_time_us) (tablet-download-pool-47c5087093214fbeae35590b4b3c3d42.queue_time_us, tablet-download-pool-47c5087093214fbeae35590b4b3c3d42.queue_time_us) (tablet-download-pool-47c5087093214fbeae35590b4b3c3d42.run_cpu_time_us, tablet-download-pool-47c5087093214fbeae35590b4b3c3d42.run_cpu_time_us) (tablet-download-pool-47c5087093214fbeae35590b4b3c3d42.run_wall_time_us, tablet-download-pool-47c5087093214fbeae35590b4b3c3d42.run_wall_time_us) (tablet-download-pool-66b8f583dfd24c2aa5a59457a8e58fbb.queue_time_us, tablet-download-pool-66b8f583dfd24c2aa5a59457a8e58fbb.queue_time_us) (tablet-download-pool-66b8f583dfd24c2aa5a59457a8e58fbb.run_cpu_time_us, tablet-download-pool-66b8f583dfd24c2aa5a59457a8e58fbb.run_cpu_time_us) (tablet-download-pool-66b8f583dfd24c2aa5a59457a8e58fbb.run_wall_time_us, tablet-download-pool-66b8f583dfd24c2aa5a59457a8e58fbb.run_wall_time_us) (tablet-download-pool-73efeba3d3094ef387e4efb03cff9d4e.queue_time_us, tablet-download-pool-73efeba3d3094ef387e4efb03cff9d4e.queue_time_us) (tablet-download-pool-73efeba3d3094ef387e4efb03cff9d4e.run_cpu_time_us, tablet-download-pool-73efeba3d3094ef387e4efb03cff9d4e.run_cpu_time_us) (tablet-doI20240502 20:46:17.820317 12055 tablet_bootstrap.cc:492] T c60371165de94a5d9ec1c7181f839e5b P c3d26eafd51547acb780d4aac5ebe7cc: No bootstrap required, opened a new log
@ 0x7f4ff691c044 kudu::TraceMetrics::InternName() at ??:0
I20240502 20:46:17.875494 12055 ts_tablet_manager.cc:1367] T c60371165de94a5d9ec1c7181f839e5b P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.086s user 0.003s sys 0.000s
I20240502 20:46:17.876037 12055 raft_consensus.cc:348] T c60371165de94a5d9ec1c7181f839e5b P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:17.876183 12055 raft_consensus.cc:374] T c60371165de94a5d9ec1c7181f839e5b P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:17.876242 12055 raft_consensus.cc:729] T c60371165de94a5d9ec1c7181f839e5b P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: FOLLOWER
I20240502 20:46:17.876380 12055 consensus_queue.cc:260] T c60371165de94a5d9ec1c7181f839e5b P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:17.876715 12055 ts_tablet_manager.cc:1398] T c60371165de94a5d9ec1c7181f839e5b P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
@ 0x7f4ff6900f98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f4ff68feefc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f4ff94b177a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240502 20:46:17.889894 4229 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 20:46:17.890071 4229 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41812:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 20:46:17.892738 4229 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 4c527b25ee464801a057bbad1ac95aec: Sending DeleteTablet for 3 replicas of tablet 2a44b2af7886427db821f873e5a79151
@ 0x7f4ff94baa45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
wnload-pool-73efeba3d3094ef387e4efb03cff9d4e.run_wall_time_us, tablet-download-pool-73efeba3d3094ef387e4efb03cff9d4e.run_wall_time_us) (tablet-download-pool-7abfd61f55554042978b50761cbd6ab5.queue_time_us, tablet-download-pool-7abfd61f55554042978b50761cbd6ab5.queue_time_us) (tablet-download-pool-7abfd61f55554042978b50761cbd6ab5.run_cpu_time_us, tablet-download-pool-7abfd61f55554042978b50761cbd6ab5.run_cpu_time_us) (tablet-download-pool-7abfd61f55554042978b50761cbd6ab5.run_wall_time_us, tablet-download-pool-7abfd61f55554042978b50761cbd6ab5.run_wall_time_us) (tablet-download-pool-8565ebbd21dd4f708a0254d1ed2d6670.queue_time_us, tablet-download-pool-8565ebbd21dd4f708a0254d1ed2d6670.queue_time_us) (tablet-download-pool-8565ebbd21dd4f708a0254d1ed2d6670.run_cpu_time_us, tablet-download-pool-8565ebbd21dd4f708a0254d1ed2d6670.run_cpu_time_us) (tablet-download-pool-8565ebbd21dd4f708a0254d1ed2d6670.run_wall_time_us, tablet-download-pool-8565ebbd21dd4f708a0254d1ed2d6670.run_wall_time_us) (tablet-download-pool-c55ba788f592448d8941ff94a8cefc52.queue_time_us, tablet-download-pool-c55ba788f592448d8941ff94a8cefc52.queue_time_us) (tablet-download-pool-c55ba788f592448d8941ff94a8cefc52.run_cpu_time_us, tablet-download-pool-c55ba788f592448d8941ff94a8cefc52.run_cpu_time_us) (tablet-download-pool-c55ba788f592448d8941ff94a8cefc52.run_wall_time_us, tablet-download-pool-c55ba788f592448d8941ff94a8cefc52.run_wall_time_us) (tablet-download-pool-cedd1c79256a4f32a73b6a88244e6181.queue_time_us, tablet-download-pool-cedd1c79256a4f32a73b6a88244e6181.queue_time_us) (tablet-download-pool-d37fab152a8a44868d3160ec3057bb98.queue_time_us, tablet-download-pool-d37fab152a8a44868d3160ec3057bb98.queue_time_us) (tablet-download-pool-d37fab152a8a44868d3160ec3057bb98.run_cpu_time_us, tablet-download-pool-d37fab152a8a44868d3160ec3057bb98.run_cpu_time_us) (tablet-download-pool-d37fab152a8a44868d3160ec3057bb98.run_wall_time_us, tablet-download-pool-d37fab152a8a44868d3160ec3057bb98.run_wall_time_us) (tablet-download-pool-f09dcfea3eb347e687f3c72f4ccc6929.queue_time_us, tablet-download-pool-f09dcfea3eb347e687f3c72f4ccc6929.queue_time_us) (tablet-download-pool-f09dcfea3eb347e687f3c72f4ccc6929.run_cpu_time_us, tablet-download-pool-f09dcfea3eb347e687f3c72f4ccc6929.run_cpu_time_us) (tablet-download-pool-f09dcfea3eb347e687f3c72f4ccc6929.run_wall_time_us, tablet-download-pool-f09dcfea3eb347e687f3c72f4ccc6929.run_wall_time_us) (tablet-download-pool-f6d4a3019ba54eb8b3506b88f04a6262.queue_time_us, tablet-download-pool-f6d4a3019ba54eb8b3506b88f04a6262.queue_time_us) (tablet-download-pool-f6d4a3019ba54eb8b3506b88f04a6262.run_cpu_time_us, tablet-download-pool-f6d4a3019ba54eb8b3506b88f04a6262.run_cpu_time_us) (tablet-download-pool-f6d4a3019ba54eb8b3506b88f04a6262.run_wall_time_us, tablet-download-pool-f6d4a3019ba54eb8b3506b88f04a6262.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
I20240502 20:46:17.816269 12065 ts_tablet_manager.cc:1888] T ef4002126e8847809add182f262d62d9 P c92e6e46782849479c8a0c6d82276fb2: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.3
*** Check failure stack trace: ***
I20240502 20:46:17.818919 12000 tablet_bootstrap.cc:492] T 533e7f3c852c4fd9bd5093085a5e8e36 P c92e6e46782849479c8a0c6d82276fb2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:17.895080 6947 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a44b2af7886427db821f873e5a79151 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:17 UTC) from {username='slave'} at 127.0.0.1:45156
I20240502 20:46:17.899454 12065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef4002126e8847809add182f262d62d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:17.878648 12103 ts_tablet_manager.cc:902] T 8602022a45e1482ebf8dab9930ce8209 P c92e6e46782849479c8a0c6d82276fb2: Initiating tablet copy from peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
I20240502 20:46:17.899554 12000 tablet_bootstrap.cc:492] T 533e7f3c852c4fd9bd5093085a5e8e36 P c92e6e46782849479c8a0c6d82276fb2: Bootstrap complete.
I20240502 20:46:17.899761 12000 ts_tablet_manager.cc:1367] T 533e7f3c852c4fd9bd5093085a5e8e36 P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.096s user 0.005s sys 0.001s
I20240502 20:46:17.900171 12103 tablet_copy_client.cc:308] T 8602022a45e1482ebf8dab9930ce8209 P c92e6e46782849479c8a0c6d82276fb2: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.71:46841
I20240502 20:46:17.900342 12000 raft_consensus.cc:348] T 533e7f3c852c4fd9bd5093085a5e8e36 P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true } }
I20240502 20:46:17.900468 12000 raft_consensus.cc:729] T 533e7f3c852c4fd9bd5093085a5e8e36 P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: LEARNER
I20240502 20:46:17.900600 12000 consensus_queue.cc:260] T 533e7f3c852c4fd9bd5093085a5e8e36 P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true } }
I20240502 20:46:17.900926 12000 ts_tablet_manager.cc:1398] T 533e7f3c852c4fd9bd5093085a5e8e36 P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
@ 0x7fb7099cadcd google::LogMessage::Fail() at ??:0
I20240502 20:46:17.921468 12013 ts_tablet_manager.cc:902] T b2508a5aa1c14b63971895953e149e17 P c3d26eafd51547acb780d4aac5ebe7cc: Initiating tablet copy from peer a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867)
@ 0x7f4ff953b76f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240502 20:46:17.925784 12013 tablet_copy_client.cc:308] T b2508a5aa1c14b63971895953e149e17 P c3d26eafd51547acb780d4aac5ebe7cc: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.70:41867
I20240502 20:46:17.902174 6987 tablet_copy_service.cc:140] P a47e5c56f13843018ac584914a8d4707: Received BeginTabletCopySession request for tablet 8602022a45e1482ebf8dab9930ce8209 from peer c92e6e46782849479c8a0c6d82276fb2 ({username='slave'} at 127.30.32.65:42487)
I20240502 20:46:17.926028 6987 tablet_copy_service.cc:161] P a47e5c56f13843018ac584914a8d4707: Beginning new tablet copy session on tablet 8602022a45e1482ebf8dab9930ce8209 from peer c92e6e46782849479c8a0c6d82276fb2 at {username='slave'} at 127.30.32.65:42487: session id = c92e6e46782849479c8a0c6d82276fb2-8602022a45e1482ebf8dab9930ce8209
I20240502 20:46:17.901294 12065 tablet_copy_client.cc:783] T ef4002126e8847809add182f262d62d9 P c92e6e46782849479c8a0c6d82276fb2: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:17.926340 12065 tablet_copy_client.cc:647] T ef4002126e8847809add182f262d62d9 P c92e6e46782849479c8a0c6d82276fb2: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:17.927124 6987 tablet_copy_source_session.cc:205] T 8602022a45e1482ebf8dab9930ce8209 P a47e5c56f13843018ac584914a8d4707: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:17.928251 12103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8602022a45e1482ebf8dab9930ce8209. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:17.911787 12113 tablet_replica.cc:330] T 2a44b2af7886427db821f873e5a79151 P a47e5c56f13843018ac584914a8d4707: stopping tablet replica
I20240502 20:46:17.929147 12113 raft_consensus.cc:2229] T 2a44b2af7886427db821f873e5a79151 P a47e5c56f13843018ac584914a8d4707 [term 1 LEADER]: Raft consensus shutting down.
I20240502 20:46:17.929374 12113 raft_consensus.cc:2258] T 2a44b2af7886427db821f873e5a79151 P a47e5c56f13843018ac584914a8d4707 [term 1 FOLLOWER]: Raft consensus is shut down!
@ 0x7f4ff954f4f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240502 20:46:17.931365 12103 tablet_copy_client.cc:783] T 8602022a45e1482ebf8dab9930ce8209 P c92e6e46782849479c8a0c6d82276fb2: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:17.931586 12113 ts_tablet_manager.cc:1875] T 2a44b2af7886427db821f873e5a79151 P a47e5c56f13843018ac584914a8d4707: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 20:46:17.932051 12065 tablet_copy_client.cc:523] T ef4002126e8847809add182f262d62d9 P c92e6e46782849479c8a0c6d82276fb2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:17.933547 12065 tablet_bootstrap.cc:492] T ef4002126e8847809add182f262d62d9 P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
I20240502 20:46:17.933980 12113 ts_tablet_manager.cc:1888] T 2a44b2af7886427db821f873e5a79151 P a47e5c56f13843018ac584914a8d4707: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:17.934056 12113 log.cc:1196] T 2a44b2af7886427db821f873e5a79151 P a47e5c56f13843018ac584914a8d4707: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-6/wal/wals/2a44b2af7886427db821f873e5a79151
I20240502 20:46:17.934336 12113 ts_tablet_manager.cc:1909] T 2a44b2af7886427db821f873e5a79151 P a47e5c56f13843018ac584914a8d4707: Deleting consensus metadata
I20240502 20:46:17.936229 4222 catalog_manager.cc:4830] TS a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): tablet 2a44b2af7886427db821f873e5a79151 (table rebalancer_dynamic_table_new_name [id=c863a144af7347ea87a5e829dd2608be]) successfully deleted
I20240502 20:46:17.936578 12103 tablet_copy_client.cc:647] T 8602022a45e1482ebf8dab9930ce8209 P c92e6e46782849479c8a0c6d82276fb2: tablet copy: Starting download of 1 WAL segments...
@ 0x7f4ff953a226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
I20240502 20:46:17.940073 12103 tablet_copy_client.cc:523] T 8602022a45e1482ebf8dab9930ce8209 P c92e6e46782849479c8a0c6d82276fb2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:17.941859 12103 tablet_bootstrap.cc:492] T 8602022a45e1482ebf8dab9930ce8209 P c92e6e46782849479c8a0c6d82276fb2: Bootstrap starting.
@ 0x7f4ff9549c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20240502 20:46:17.825902 12063 tablet_bootstrap.cc:492] T 7ca68870fae3480786c84bd86dc4897e P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:46:17.901640 6723 tablet_copy_service.cc:342] P 7845c397230d49358ba2fad9db53f97c: Request end of tablet copy session c92e6e46782849479c8a0c6d82276fb2-533e7f3c852c4fd9bd5093085a5e8e36 received from {username='slave'} at 127.30.32.65:34313
I20240502 20:46:17.950134 6723 tablet_copy_service.cc:433] P 7845c397230d49358ba2fad9db53f97c: ending tablet copy session c92e6e46782849479c8a0c6d82276fb2-533e7f3c852c4fd9bd5093085a5e8e36 on tablet 533e7f3c852c4fd9bd5093085a5e8e36 with peer c92e6e46782849479c8a0c6d82276fb2
I20240502 20:46:17.950212 12063 ts_tablet_manager.cc:1367] T 7ca68870fae3480786c84bd86dc4897e P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.145s user 0.002s sys 0.000s
I20240502 20:46:17.950732 12063 raft_consensus.cc:348] T 7ca68870fae3480786c84bd86dc4897e P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:17.950846 12063 raft_consensus.cc:374] T 7ca68870fae3480786c84bd86dc4897e P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:17.950873 12063 raft_consensus.cc:729] T 7ca68870fae3480786c84bd86dc4897e P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:46:17.841817 11986 ts_tablet_manager.cc:902] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P 923cfab926e7443f8ffd7b21ad0a9d91: Initiating tablet copy from peer ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:46:17.951000 12063 consensus_queue.cc:260] T 7ca68870fae3480786c84bd86dc4897e P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:17.907377 12118 ts_tablet_manager.cc:902] T 3ba2f5d327f3403ea1604aae9485b960 P 923cfab926e7443f8ffd7b21ad0a9d91: Initiating tablet copy from peer ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:46:17.951313 12063 ts_tablet_manager.cc:1398] T 7ca68870fae3480786c84bd86dc4897e P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:17.951537 11986 tablet_copy_client.cc:308] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P 923cfab926e7443f8ffd7b21ad0a9d91: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.68:38193
@ 0x7fb7099ceb93 google::LogMessage::SendToLog() at ??:0
I20240502 20:46:17.951720 12118 tablet_copy_client.cc:308] T 3ba2f5d327f3403ea1604aae9485b960 P 923cfab926e7443f8ffd7b21ad0a9d91: tablet copy: Beginning tablet copy session from remote peer at address 127.30.32.68:38193
@ 0x55b80842d75c std::function<>::operator()() at ??:0
@ 0x7fb7099ca7cc google::LogMessage::Flush() at ??:0
@ 0x7f4ff6903e06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f4ff69046ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
I20240502 20:46:17.961835 4790 tablet_copy_service.cc:140] P ed47c023452a497192f35d2f617bf9bd: Received BeginTabletCopySession request for tablet 77b294bcc497479e8e8bd8e7f9cc1ff2 from peer 923cfab926e7443f8ffd7b21ad0a9d91 ({username='slave'} at 127.30.32.67:42459)
I20240502 20:46:17.961973 4790 tablet_copy_service.cc:161] P ed47c023452a497192f35d2f617bf9bd: Beginning new tablet copy session on tablet 77b294bcc497479e8e8bd8e7f9cc1ff2 from peer 923cfab926e7443f8ffd7b21ad0a9d91 at {username='slave'} at 127.30.32.67:42459: session id = 923cfab926e7443f8ffd7b21ad0a9d91-77b294bcc497479e8e8bd8e7f9cc1ff2
I20240502 20:46:17.962523 12103 tablet_bootstrap.cc:492] T 8602022a45e1482ebf8dab9930ce8209 P c92e6e46782849479c8a0c6d82276fb2: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:17.962826 12103 tablet_bootstrap.cc:492] T 8602022a45e1482ebf8dab9930ce8209 P c92e6e46782849479c8a0c6d82276fb2: Bootstrap complete.
@ 0x7f4ff690603b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20240502 20:46:17.963130 4789 tablet_copy_service.cc:140] P ed47c023452a497192f35d2f617bf9bd: Received BeginTabletCopySession request for tablet 3ba2f5d327f3403ea1604aae9485b960 from peer 923cfab926e7443f8ffd7b21ad0a9d91 ({username='slave'} at 127.30.32.67:42459)
I20240502 20:46:17.963217 4789 tablet_copy_service.cc:161] P ed47c023452a497192f35d2f617bf9bd: Beginning new tablet copy session on tablet 3ba2f5d327f3403ea1604aae9485b960 from peer 923cfab926e7443f8ffd7b21ad0a9d91 at {username='slave'} at 127.30.32.67:42459: session id = 923cfab926e7443f8ffd7b21ad0a9d91-3ba2f5d327f3403ea1604aae9485b960
I20240502 20:46:17.963275 4790 tablet_copy_source_session.cc:205] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P ed47c023452a497192f35d2f617bf9bd: Tablet Copy: opened 0 blocks and 1 log segments
@ 0x55b80842d75c std::function<>::operator()() at ??:0
I20240502 20:46:17.964148 4789 tablet_copy_source_session.cc:205] T 3ba2f5d327f3403ea1604aae9485b960 P ed47c023452a497192f35d2f617bf9bd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:17.964511 12103 ts_tablet_manager.cc:1367] T 8602022a45e1482ebf8dab9930ce8209 P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.003s
I20240502 20:46:17.965242 12103 raft_consensus.cc:348] T 8602022a45e1482ebf8dab9930ce8209 P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true } }
I20240502 20:46:17.965384 12103 raft_consensus.cc:729] T 8602022a45e1482ebf8dab9930ce8209 P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: LEARNER
I20240502 20:46:17.826674 12064 tablet_bootstrap.cc:492] T c60371165de94a5d9ec1c7181f839e5b P a4bf1df7fa6b4948be502c182bfbb833: No bootstrap required, opened a new log
I20240502 20:46:17.994616 4229 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41812:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 20:46:18.015880 12064 ts_tablet_manager.cc:1367] T c60371165de94a5d9ec1c7181f839e5b P a4bf1df7fa6b4948be502c182bfbb833: Time spent bootstrapping tablet: real 0.209s user 0.003s sys 0.000s
I20240502 20:46:18.016469 12064 raft_consensus.cc:348] T c60371165de94a5d9ec1c7181f839e5b P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:18.016611 12064 raft_consensus.cc:374] T c60371165de94a5d9ec1c7181f839e5b P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:18.016645 12064 raft_consensus.cc:729] T c60371165de94a5d9ec1c7181f839e5b P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Initialized, Role: FOLLOWER
I20240502 20:46:18.016777 12064 consensus_queue.cc:260] T c60371165de94a5d9ec1c7181f839e5b P a4bf1df7fa6b4948be502c182bfbb833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
@ 0x7fb7099cbf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240502 20:46:18.017184 12064 ts_tablet_manager.cc:1398] T c60371165de94a5d9ec1c7181f839e5b P a4bf1df7fa6b4948be502c182bfbb833: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:46:18.017406 12064 tablet_bootstrap.cc:492] T 7ca68870fae3480786c84bd86dc4897e P a4bf1df7fa6b4948be502c182bfbb833: Bootstrap starting.
I20240502 20:46:18.018736 12064 tablet_bootstrap.cc:654] T 7ca68870fae3480786c84bd86dc4897e P a4bf1df7fa6b4948be502c182bfbb833: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:17.893486 6802 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a44b2af7886427db821f873e5a79151 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 20:46:17 UTC) from {username='slave'} at 127.0.0.1:35290
I20240502 20:46:18.020010 12064 tablet_bootstrap.cc:492] T 7ca68870fae3480786c84bd86dc4897e P a4bf1df7fa6b4948be502c182bfbb833: No bootstrap required, opened a new log
@ 0x7fb70ae34044 kudu::TraceMetrics::InternName() at ??:0
I20240502 20:46:18.020084 12064 ts_tablet_manager.cc:1367] T 7ca68870fae3480786c84bd86dc4897e P a4bf1df7fa6b4948be502c182bfbb833: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:46:18.020475 12064 raft_consensus.cc:348] T 7ca68870fae3480786c84bd86dc4897e P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:18.020560 12064 raft_consensus.cc:374] T 7ca68870fae3480786c84bd86dc4897e P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:18.020586 12064 raft_consensus.cc:729] T 7ca68870fae3480786c84bd86dc4897e P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Initialized, Role: FOLLOWER
I20240502 20:46:18.020787 12064 consensus_queue.cc:260] T 7ca68870fae3480786c84bd86dc4897e P a4bf1df7fa6b4948be502c182bfbb833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:17.929589 6848 tablet_copy_service.cc:140] P a4bf1df7fa6b4948be502c182bfbb833: Received BeginTabletCopySession request for tablet b2508a5aa1c14b63971895953e149e17 from peer c3d26eafd51547acb780d4aac5ebe7cc ({username='slave'} at 127.30.32.66:55279)
I20240502 20:46:18.021221 6848 tablet_copy_service.cc:161] P a4bf1df7fa6b4948be502c182bfbb833: Beginning new tablet copy session on tablet b2508a5aa1c14b63971895953e149e17 from peer c3d26eafd51547acb780d4aac5ebe7cc at {username='slave'} at 127.30.32.66:55279: session id = c3d26eafd51547acb780d4aac5ebe7cc-b2508a5aa1c14b63971895953e149e17
I20240502 20:46:18.022157 6848 tablet_copy_source_session.cc:205] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833: Tablet Copy: opened 0 blocks and 1 log segments
@ 0x7fb70ae18f98 kudu::ThreadPool::ThreadPool() at ??:0
I20240502 20:46:18.028625 6947 tablet_service.cc:1467] Processing CreateTablet for tablet 9ede0f8eef3440349ec5f0b93878c1f4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c530c1854d6d454a9d81d811a20ce8a9]), partition=
I20240502 20:46:18.028409 4750 tablet_service.cc:1467] Processing CreateTablet for tablet 9ede0f8eef3440349ec5f0b93878c1f4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c530c1854d6d454a9d81d811a20ce8a9]), partition=
I20240502 20:46:18.035177 6947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ede0f8eef3440349ec5f0b93878c1f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:18.035406 4750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ede0f8eef3440349ec5f0b93878c1f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:18.023239 12013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2508a5aa1c14b63971895953e149e17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:18.037387 12063 tablet_bootstrap.cc:492] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd: Bootstrap starting.
I20240502 20:46:18.037699 12056 tablet_bootstrap.cc:492] T 9ede0f8eef3440349ec5f0b93878c1f4 P a47e5c56f13843018ac584914a8d4707: Bootstrap starting.
I20240502 20:46:18.038328 12063 tablet_bootstrap.cc:654] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:18.038590 12056 tablet_bootstrap.cc:654] T 9ede0f8eef3440349ec5f0b93878c1f4 P a47e5c56f13843018ac584914a8d4707: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:18.039799 12013 tablet_copy_client.cc:783] T b2508a5aa1c14b63971895953e149e17 P c3d26eafd51547acb780d4aac5ebe7cc: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:18.039942 12013 tablet_copy_client.cc:647] T b2508a5aa1c14b63971895953e149e17 P c3d26eafd51547acb780d4aac5ebe7cc: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:18.041461 12013 tablet_copy_client.cc:523] T b2508a5aa1c14b63971895953e149e17 P c3d26eafd51547acb780d4aac5ebe7cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:18.041623 12063 tablet_bootstrap.cc:492] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd: No bootstrap required, opened a new log
I20240502 20:46:18.041724 12063 ts_tablet_manager.cc:1367] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 20:46:18.042169 12063 raft_consensus.cc:348] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:18.042291 12063 raft_consensus.cc:374] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:18.042333 12063 raft_consensus.cc:729] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Initialized, Role: FOLLOWER
I20240502 20:46:18.042464 12063 consensus_queue.cc:260] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:18.042737 12013 tablet_bootstrap.cc:492] T b2508a5aa1c14b63971895953e149e17 P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap starting.
I20240502 20:46:18.042754 12063 ts_tablet_manager.cc:1398] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
@ 0x7fb70ae16efc kudu::ThreadPoolBuilder::Build() at ??:0
I20240502 20:46:18.049247 12013 tablet_bootstrap.cc:492] T b2508a5aa1c14b63971895953e149e17 P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:18.071862 12013 tablet_bootstrap.cc:492] T b2508a5aa1c14b63971895953e149e17 P c3d26eafd51547acb780d4aac5ebe7cc: Bootstrap complete.
I20240502 20:46:18.072104 12013 ts_tablet_manager.cc:1367] T b2508a5aa1c14b63971895953e149e17 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.000s
I20240502 20:46:18.072774 12013 raft_consensus.cc:348] T b2508a5aa1c14b63971895953e149e17 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } }
I20240502 20:46:18.072957 12013 raft_consensus.cc:729] T b2508a5aa1c14b63971895953e149e17 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Initialized, Role: LEARNER
I20240502 20:46:18.073110 12013 consensus_queue.cc:260] T b2508a5aa1c14b63971895953e149e17 P c3d26eafd51547acb780d4aac5ebe7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } }
I20240502 20:46:17.965483 12103 consensus_queue.cc:260] T 8602022a45e1482ebf8dab9930ce8209 P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true } }
I20240502 20:46:17.969569 12065 tablet_bootstrap.cc:492] T ef4002126e8847809add182f262d62d9 P c92e6e46782849479c8a0c6d82276fb2: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:18.073976 12103 ts_tablet_manager.cc:1398] T 8602022a45e1482ebf8dab9930ce8209 P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.109s user 0.001s sys 0.000s
I20240502 20:46:18.074033 12065 tablet_bootstrap.cc:492] T ef4002126e8847809add182f262d62d9 P c92e6e46782849479c8a0c6d82276fb2: Bootstrap complete.
@ 0x7f4ff68f3807 kudu::Thread::SuperviseThread() at ??:0
I20240502 20:46:18.074268 12065 ts_tablet_manager.cc:1367] T ef4002126e8847809add182f262d62d9 P c92e6e46782849479c8a0c6d82276fb2: Time spent bootstrapping tablet: real 0.141s user 0.005s sys 0.002s
I20240502 20:46:18.074882 12065 raft_consensus.cc:348] T ef4002126e8847809add182f262d62d9 P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true } }
I20240502 20:46:18.075023 12065 raft_consensus.cc:729] T ef4002126e8847809add182f262d62d9 P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c92e6e46782849479c8a0c6d82276fb2, State: Initialized, Role: LEARNER
I20240502 20:46:18.075146 12065 consensus_queue.cc:260] T ef4002126e8847809add182f262d62d9 P c92e6e46782849479c8a0c6d82276fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true } }
I20240502 20:46:18.075547 12065 ts_tablet_manager.cc:1398] T ef4002126e8847809add182f262d62d9 P c92e6e46782849479c8a0c6d82276fb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
@ 0x7f4ff71316db start_thread at ??:0
I20240502 20:46:17.968788 12118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ba2f5d327f3403ea1604aae9485b960. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:18.077211 11986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77b294bcc497479e8e8bd8e7f9cc1ff2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:18.079586 12118 tablet_copy_client.cc:783] T 3ba2f5d327f3403ea1604aae9485b960 P 923cfab926e7443f8ffd7b21ad0a9d91: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:18.079774 12118 tablet_copy_client.cc:647] T 3ba2f5d327f3403ea1604aae9485b960 P 923cfab926e7443f8ffd7b21ad0a9d91: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:18.082396 4638 raft_consensus.cc:1203] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 20:46:18.082954 6986 tablet_copy_service.cc:342] P a47e5c56f13843018ac584914a8d4707: Request end of tablet copy session c92e6e46782849479c8a0c6d82276fb2-8602022a45e1482ebf8dab9930ce8209 received from {username='slave'} at 127.30.32.65:42487
I20240502 20:46:18.083057 6986 tablet_copy_service.cc:433] P a47e5c56f13843018ac584914a8d4707: ending tablet copy session c92e6e46782849479c8a0c6d82276fb2-8602022a45e1482ebf8dab9930ce8209 on tablet 8602022a45e1482ebf8dab9930ce8209 with peer c92e6e46782849479c8a0c6d82276fb2
I20240502 20:46:18.083251 6987 tablet_copy_service.cc:342] P a47e5c56f13843018ac584914a8d4707: Request end of tablet copy session c92e6e46782849479c8a0c6d82276fb2-ef4002126e8847809add182f262d62d9 received from {username='slave'} at 127.30.32.65:42487
I20240502 20:46:18.083326 6987 tablet_copy_service.cc:433] P a47e5c56f13843018ac584914a8d4707: ending tablet copy session c92e6e46782849479c8a0c6d82276fb2-ef4002126e8847809add182f262d62d9 on tablet ef4002126e8847809add182f262d62d9 with peer c92e6e46782849479c8a0c6d82276fb2
@ 0x7fb70d9c977a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240502 20:46:18.079581 11986 tablet_copy_client.cc:783] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P 923cfab926e7443f8ffd7b21ad0a9d91: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:18.085757 11986 tablet_copy_client.cc:647] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P 923cfab926e7443f8ffd7b21ad0a9d91: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:18.073508 12013 ts_tablet_manager.cc:1398] T b2508a5aa1c14b63971895953e149e17 P c3d26eafd51547acb780d4aac5ebe7cc: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 20:46:18.086849 11986 tablet_copy_client.cc:523] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P 923cfab926e7443f8ffd7b21ad0a9d91: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:18.088253 11986 tablet_bootstrap.cc:492] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
I20240502 20:46:18.089310 12118 tablet_copy_client.cc:523] T 3ba2f5d327f3403ea1604aae9485b960 P 923cfab926e7443f8ffd7b21ad0a9d91: tablet copy: Tablet Copy complete. Replacing tablet superblock.
@ 0x7f4ff466971f clone at ??:0
I20240502 20:46:18.091293 12118 tablet_bootstrap.cc:492] T 3ba2f5d327f3403ea1604aae9485b960 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap starting.
@ 0x7fb70d9d2a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7fb70da5376f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7fb70da674f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240502 20:46:18.022514 12064 ts_tablet_manager.cc:1398] T 7ca68870fae3480786c84bd86dc4897e P a4bf1df7fa6b4948be502c182bfbb833: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:18.030387 12160 tablet_replica.cc:330] T 2a44b2af7886427db821f873e5a79151 P a4bf1df7fa6b4948be502c182bfbb833: stopping tablet replica
I20240502 20:46:18.102645 12160 raft_consensus.cc:2229] T 2a44b2af7886427db821f873e5a79151 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:18.102871 12160 raft_consensus.cc:2258] T 2a44b2af7886427db821f873e5a79151 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:18.087237 6848 tablet_copy_service.cc:342] P a4bf1df7fa6b4948be502c182bfbb833: Request end of tablet copy session c3d26eafd51547acb780d4aac5ebe7cc-b2508a5aa1c14b63971895953e149e17 received from {username='slave'} at 127.30.32.66:55279
I20240502 20:46:18.103040 6848 tablet_copy_service.cc:433] P a4bf1df7fa6b4948be502c182bfbb833: ending tablet copy session c3d26eafd51547acb780d4aac5ebe7cc-b2508a5aa1c14b63971895953e149e17 on tablet b2508a5aa1c14b63971895953e149e17 with peer c3d26eafd51547acb780d4aac5ebe7cc
I20240502 20:46:18.105562 12160 ts_tablet_manager.cc:1875] T 2a44b2af7886427db821f873e5a79151 P a4bf1df7fa6b4948be502c182bfbb833: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7fb70da52226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
W20240502 20:46:18.107610 6900 connection.cc:535] client connection to 127.30.32.72:44643 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 20:46:18.107622 4703 connection.cc:535] client connection to 127.30.32.72:44643 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 20:46:18.107683 4572 connection.cc:535] client connection to 127.30.32.72:44643 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 20:46:18.107749 6900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 20:46:18.107772 4703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 20:46:18.107805 4572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 20:46:18.108256 4310 connection.cc:535] client connection to 127.30.32.72:44643 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 20:46:18.108295 12160 ts_tablet_manager.cc:1888] T 2a44b2af7886427db821f873e5a79151 P a4bf1df7fa6b4948be502c182bfbb833: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 20:46:18.108356 12160 log.cc:1196] T 2a44b2af7886427db821f873e5a79151 P a4bf1df7fa6b4948be502c182bfbb833: Deleting WAL directory at /tmp/dist-test-taskolF7xA/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714682724551884-30849-0/raft_consensus-itest-cluster/ts-5/wal/wals/2a44b2af7886427db821f873e5a79151
W20240502 20:46:18.108387 4310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 20:46:18.108901 4441 connection.cc:535] client connection to 127.30.32.72:44643 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 20:46:18.109036 4441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 20:46:18.109143 11986 tablet_bootstrap.cc:492] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:18.109366 11986 tablet_bootstrap.cc:492] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap complete.
I20240502 20:46:18.109506 11986 ts_tablet_manager.cc:1367] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.004s
W20240502 20:46:18.109537 6768 connection.cc:535] client connection to 127.30.32.72:44643 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 20:46:18.109639 6768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 20:46:18.110109 4219 connection.cc:535] client connection to 127.30.32.72:44643 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 20:46:18.110116 11986 raft_consensus.cc:348] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } }
I20240502 20:46:18.110229 11986 raft_consensus.cc:729] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: LEARNER
W20240502 20:46:18.110234 4219 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 20:46:18.110312 11986 consensus_queue.cc:260] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } }
I20240502 20:46:18.110373 12160 ts_tablet_manager.cc:1909] T 2a44b2af7886427db821f873e5a79151 P a4bf1df7fa6b4948be502c182bfbb833: Deleting consensus metadata
I20240502 20:46:18.111281 11986 ts_tablet_manager.cc:1398] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 20:46:18.111680 4789 tablet_copy_service.cc:342] P ed47c023452a497192f35d2f617bf9bd: Request end of tablet copy session 923cfab926e7443f8ffd7b21ad0a9d91-77b294bcc497479e8e8bd8e7f9cc1ff2 received from {username='slave'} at 127.30.32.67:42459
I20240502 20:46:18.111763 4789 tablet_copy_service.cc:433] P ed47c023452a497192f35d2f617bf9bd: ending tablet copy session 923cfab926e7443f8ffd7b21ad0a9d91-77b294bcc497479e8e8bd8e7f9cc1ff2 on tablet 77b294bcc497479e8e8bd8e7f9cc1ff2 with peer 923cfab926e7443f8ffd7b21ad0a9d91
I20240502 20:46:18.112332 4220 catalog_manager.cc:4830] TS a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867): tablet 2a44b2af7886427db821f873e5a79151 (table rebalancer_dynamic_table_new_name [id=c863a144af7347ea87a5e829dd2608be]) successfully deleted
I20240502 20:46:18.115494 12118 tablet_bootstrap.cc:492] T 3ba2f5d327f3403ea1604aae9485b960 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:18.115780 12118 tablet_bootstrap.cc:492] T 3ba2f5d327f3403ea1604aae9485b960 P 923cfab926e7443f8ffd7b21ad0a9d91: Bootstrap complete.
I20240502 20:46:18.116001 12118 ts_tablet_manager.cc:1367] T 3ba2f5d327f3403ea1604aae9485b960 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.000s
I20240502 20:46:18.116516 12118 raft_consensus.cc:348] T 3ba2f5d327f3403ea1604aae9485b960 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } }
I20240502 20:46:18.116644 12118 raft_consensus.cc:729] T 3ba2f5d327f3403ea1604aae9485b960 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Initialized, Role: LEARNER
I20240502 20:46:18.117739 12118 consensus_queue.cc:260] T 3ba2f5d327f3403ea1604aae9485b960 P 923cfab926e7443f8ffd7b21ad0a9d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } }
I20240502 20:46:18.118072 12118 ts_tablet_manager.cc:1398] T 3ba2f5d327f3403ea1604aae9485b960 P 923cfab926e7443f8ffd7b21ad0a9d91: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:18.119410 4789 tablet_copy_service.cc:342] P ed47c023452a497192f35d2f617bf9bd: Request end of tablet copy session 923cfab926e7443f8ffd7b21ad0a9d91-3ba2f5d327f3403ea1604aae9485b960 received from {username='slave'} at 127.30.32.67:42459
I20240502 20:46:18.119488 4789 tablet_copy_service.cc:433] P ed47c023452a497192f35d2f617bf9bd: ending tablet copy session 923cfab926e7443f8ffd7b21ad0a9d91-3ba2f5d327f3403ea1604aae9485b960 on tablet 3ba2f5d327f3403ea1604aae9485b960 with peer 923cfab926e7443f8ffd7b21ad0a9d91
W20240502 20:46:18.120106 4310 consensus_peers.cc:485] T a7ea68e8581a4cd5a6212abc96dfbcde P c92e6e46782849479c8a0c6d82276fb2 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:18.120170 4310 consensus_peers.cc:485] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:18.122545 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:18.122606 4572 consensus_peers.cc:485] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:18.123282 4703 consensus_peers.cc:485] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:18.123359 4703 consensus_peers.cc:485] T 63f98288c8b1483493d5018c36e611bb P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:18.124151 4703 consensus_peers.cc:485] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
@ 0x7fb70da61c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x564b9b7a875c std::function<>::operator()() at ??:0
@ 0x7fb70ae1be06 kudu::ThreadPool::DispatchThread() at ??:0
W20240502 20:46:18.132752 4219 catalog_manager.cc:4573] TS 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2a44b2af7886427db821f873e5a79151: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:18.132432 6900 consensus_peers.cc:485] T 8bbf33f0ec4a4adb9a979b08606ee822 P a47e5c56f13843018ac584914a8d4707 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:18.154918 4310 consensus_peers.cc:485] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:18.177065 6900 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.32.72:44643, user_credentials={real_user=slave}} blocked reactor thread for 44647.3us
W20240502 20:46:18.177314 6900 consensus_peers.cc:485] T 3081e646f1b044afac5f8e1e8d45aa6b P a47e5c56f13843018ac584914a8d4707 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
@ 0x7fb70ae1c6ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fb70ae1e03b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x564b9b7a875c std::function<>::operator()() at ??:0
@ 0x7fb70ae0b807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb70b6496db start_thread at ??:0
@ 0x7fb708b8171f clone at ??:0
W20240502 20:46:18.134159 4441 consensus_peers.cc:485] T 6c8f977981c54fa1bf5d173e83608146 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:18.180509 4508 raft_consensus.cc:1203] T a999f994b78a4b33a95ee7d5407cba20 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240502 20:46:18.195991 4441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.32.72:44643, user_credentials={real_user=slave}} blocked reactor thread for 61840.9us
W20240502 20:46:18.196251 4441 consensus_peers.cc:485] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:18.169548 4703 consensus_peers.cc:485] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:18.130460 6768 consensus_peers.cc:485] T 6201083694c344c8ad63720be89a3cdb P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:18.196796 6768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.32.72:44643, user_credentials={real_user=slave}} blocked reactor thread for 66355.4us
I20240502 20:46:18.134878 11605 raft_consensus.cc:482] T 7ca68870fae3480786c84bd86dc4897e P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:18.197409 11605 raft_consensus.cc:504] T 7ca68870fae3480786c84bd86dc4897e P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:18.197861 11605 leader_election.cc:288] T 7ca68870fae3480786c84bd86dc4897e P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:46:18.198613 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ca68870fae3480786c84bd86dc4897e" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd" is_pre_election: true
I20240502 20:46:18.198774 4768 raft_consensus.cc:2454] T 7ca68870fae3480786c84bd86dc4897e P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 0.
I20240502 20:46:18.199499 6771 leader_election.cc:302] T 7ca68870fae3480786c84bd86dc4897e P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4bf1df7fa6b4948be502c182bfbb833, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:46:18.199689 11863 raft_consensus.cc:2781] T 7ca68870fae3480786c84bd86dc4897e P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:18.199743 11863 raft_consensus.cc:482] T 7ca68870fae3480786c84bd86dc4897e P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:18.199788 11863 raft_consensus.cc:3037] T 7ca68870fae3480786c84bd86dc4897e P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:18.201071 11863 raft_consensus.cc:504] T 7ca68870fae3480786c84bd86dc4897e P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:18.201534 11863 leader_election.cc:288] T 7ca68870fae3480786c84bd86dc4897e P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 election: Requested vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:46:18.202203 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ca68870fae3480786c84bd86dc4897e" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd"
I20240502 20:46:18.202327 4768 raft_consensus.cc:3037] T 7ca68870fae3480786c84bd86dc4897e P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:18.203506 4768 raft_consensus.cc:2454] T 7ca68870fae3480786c84bd86dc4897e P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 1.
I20240502 20:46:18.204386 6771 leader_election.cc:302] T 7ca68870fae3480786c84bd86dc4897e P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4bf1df7fa6b4948be502c182bfbb833, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:46:18.204564 10776 raft_consensus.cc:2781] T 7ca68870fae3480786c84bd86dc4897e P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:18.204635 10776 raft_consensus.cc:686] T 7ca68870fae3480786c84bd86dc4897e P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEADER]: Becoming Leader. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Running, Role: LEADER
I20240502 20:46:18.204779 10776 consensus_queue.cc:237] T 7ca68870fae3480786c84bd86dc4897e P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
W20240502 20:46:18.206609 6768 consensus_peers.cc:485] T 52e302882311488ab1cac90ad03fe270 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:18.206698 6768 consensus_peers.cc:485] T 71537846373844cc9ef396b79af60db4 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:18.208099 11605 raft_consensus.cc:482] T c60371165de94a5d9ec1c7181f839e5b P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:18.208195 11605 raft_consensus.cc:504] T c60371165de94a5d9ec1c7181f839e5b P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:18.208475 4231 catalog_manager.cc:5484] T 7ca68870fae3480786c84bd86dc4897e P a4bf1df7fa6b4948be502c182bfbb833 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70). New cstate: current_term: 1 leader_uuid: "a4bf1df7fa6b4948be502c182bfbb833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:18.208636 11605 leader_election.cc:288] T c60371165de94a5d9ec1c7181f839e5b P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643), c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097)
I20240502 20:46:18.209133 4508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c60371165de94a5d9ec1c7181f839e5b" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" is_pre_election: true
I20240502 20:46:18.209285 4508 raft_consensus.cc:2454] T c60371165de94a5d9ec1c7181f839e5b P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 0.
I20240502 20:46:18.209697 6770 leader_election.cc:302] T c60371165de94a5d9ec1c7181f839e5b P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4bf1df7fa6b4948be502c182bfbb833, c3d26eafd51547acb780d4aac5ebe7cc; no voters:
I20240502 20:46:18.209882 11605 raft_consensus.cc:2781] T c60371165de94a5d9ec1c7181f839e5b P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:18.209937 11605 raft_consensus.cc:482] T c60371165de94a5d9ec1c7181f839e5b P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:18.209981 11605 raft_consensus.cc:3037] T c60371165de94a5d9ec1c7181f839e5b P a4bf1df7fa6b4948be502c182bfbb833 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:18.211059 11605 raft_consensus.cc:504] T c60371165de94a5d9ec1c7181f839e5b P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:18.211390 11605 leader_election.cc:288] T c60371165de94a5d9ec1c7181f839e5b P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 election: Requested vote from peers 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643), c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097)
I20240502 20:46:18.212821 4508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c60371165de94a5d9ec1c7181f839e5b" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc"
I20240502 20:46:18.212925 4508 raft_consensus.cc:3037] T c60371165de94a5d9ec1c7181f839e5b P c3d26eafd51547acb780d4aac5ebe7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:18.213928 4508 raft_consensus.cc:2454] T c60371165de94a5d9ec1c7181f839e5b P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 1.
I20240502 20:46:18.214928 6770 leader_election.cc:302] T c60371165de94a5d9ec1c7181f839e5b P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4bf1df7fa6b4948be502c182bfbb833, c3d26eafd51547acb780d4aac5ebe7cc; no voters:
I20240502 20:46:18.215073 10776 raft_consensus.cc:2781] T c60371165de94a5d9ec1c7181f839e5b P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:18.215137 10776 raft_consensus.cc:686] T c60371165de94a5d9ec1c7181f839e5b P a4bf1df7fa6b4948be502c182bfbb833 [term 1 LEADER]: Becoming Leader. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Running, Role: LEADER
I20240502 20:46:18.215245 10776 consensus_queue.cc:237] T c60371165de94a5d9ec1c7181f839e5b P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
W20240502 20:46:18.217393 6768 consensus_peers.cc:485] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:18.218456 6768 leader_election.cc:334] T c60371165de94a5d9ec1c7181f839e5b P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:18.218552 6768 leader_election.cc:334] T c60371165de94a5d9ec1c7181f839e5b P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
I20240502 20:46:18.218737 4231 catalog_manager.cc:5484] T c60371165de94a5d9ec1c7181f839e5b P a4bf1df7fa6b4948be502c182bfbb833 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70). New cstate: current_term: 1 leader_uuid: "a4bf1df7fa6b4948be502c182bfbb833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: UNKNOWN } } }
W20240502 20:46:18.235806 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
I20240502 20:46:18.250669 4638 raft_consensus.cc:1203] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 20:46:18.300566 9125 raft_consensus.cc:482] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:18.300710 9125 raft_consensus.cc:504] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:18.301303 9125 leader_election.cc:288] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
W20240502 20:46:18.302825 4703 leader_election.cc:334] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:18.307660 4572 consensus_peers.cc:485] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:18.333843 4507 raft_consensus.cc:1203] T b2508a5aa1c14b63971895953e149e17 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240502 20:46:18.354385 4312 connection.cc:535] client connection to 127.30.32.69:42429 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 20:46:18.354637 4443 connection.cc:535] client connection to 127.30.32.69:42429 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 20:46:18.354637 4574 connection.cc:535] client connection to 127.30.32.69:42429 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 20:46:18.356159 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:18.356734 6770 connection.cc:535] client connection to 127.30.32.69:42429 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 20:46:18.357131 4574 consensus_peers.cc:485] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:18.357283 4312 consensus_peers.cc:485] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:18.357755 6770 consensus_peers.cc:485] T 71537846373844cc9ef396b79af60db4 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:18.357841 6770 consensus_peers.cc:485] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:18.357894 6770 leader_election.cc:334] T 7ca68870fae3480786c84bd86dc4897e P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:18.358018 6770 leader_election.cc:334] T 7ca68870fae3480786c84bd86dc4897e P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:18.359424 4443 consensus_peers.cc:485] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:18.379319 6768 consensus_peers.cc:485] T d37730b109d04b6395f75c7552524877 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:18.387813 4443 consensus_peers.cc:485] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:18.398885 4705 consensus_peers.cc:485] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:18.402577 4312 consensus_peers.cc:485] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:18.403709 4441 consensus_peers.cc:485] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:18.418557 4312 consensus_peers.cc:485] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:18.418713 4443 consensus_peers.cc:485] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:18.419492 4706 connection.cc:535] client connection to 127.30.32.71:46841 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 20:46:18.420089 4444 connection.cc:535] client connection to 127.30.32.71:46841 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 20:46:18.421193 4443 connection.cc:535] server connection from 127.30.32.71:46593 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 20:46:18.421413 6771 connection.cc:535] client connection to 127.30.32.71:46841 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 20:46:18.421500 4575 connection.cc:535] client connection to 127.30.32.71:46841 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 20:46:18.422277 12198 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 20:46:18.420600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.32.71:46841 (local address 127.30.32.66:53577)
0502 20:46:18.420726 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0502 20:46:18.420739 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0502 20:46:18.420823 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 20:46:18.421681 (+ 858us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":49}
W20240502 20:46:18.422312 6771 consensus_peers.cc:485] T a9afb75bfc114f59850eaea084a306f0 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:18.422384 6771 consensus_peers.cc:485] T 6201083694c344c8ad63720be89a3cdb P a4bf1df7fa6b4948be502c182bfbb833 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:18.422503 4444 consensus_peers.cc:485] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:18.422574 4444 consensus_peers.cc:485] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:18.422859 4575 consensus_peers.cc:485] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:18.423043 12189 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 20:46:18.420025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.32.71:46841 (local address 127.30.32.68:38415)
0502 20:46:18.420258 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0502 20:46:18.420270 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0502 20:46:18.420362 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 20:46:18.422662 (+ 2300us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":150}
W20240502 20:46:18.423184 4706 leader_election.cc:334] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 20:46:18.423285 4706 leader_election.cc:302] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed47c023452a497192f35d2f617bf9bd; no voters: 201d2b6485084e7dad2cbb722af94779, a47e5c56f13843018ac584914a8d4707
W20240502 20:46:18.423398 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:18.423466 12216 raft_consensus.cc:2726] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 20:46:18.423476 4706 consensus_peers.cc:485] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:18.423552 4706 consensus_peers.cc:485] T 3ba2f5d327f3403ea1604aae9485b960 P ed47c023452a497192f35d2f617bf9bd -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:18.476531 4575 consensus_peers.cc:485] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:18.522485 4638 raft_consensus.cc:1203] T 3ba2f5d327f3403ea1604aae9485b960 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240502 20:46:18.523337 4443 consensus_peers.cc:485] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:18.525206 4705 consensus_peers.cc:485] T 63f98288c8b1483493d5018c36e611bb P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:18.542263 4575 consensus_peers.cc:485] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:18.568753 4574 consensus_peers.cc:485] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:18.582239 4574 consensus_peers.cc:485] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:18.583242 4575 consensus_peers.cc:485] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:18.589521 10776 consensus_queue.cc:1035] T 7ca68870fae3480786c84bd86dc4897e P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 20:46:18.590015 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:18.592321 6770 consensus_peers.cc:485] T dff789a7a50c4ff0a1eb29489ac7901a P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:18.592945 6770 consensus_peers.cc:485] T 7ca68870fae3480786c84bd86dc4897e P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:18.625314 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:18.632799 6771 consensus_peers.cc:485] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:18.636543 10776 consensus_queue.cc:1035] T c60371165de94a5d9ec1c7181f839e5b P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 20:46:18.639642 6768 consensus_peers.cc:485] T c60371165de94a5d9ec1c7181f839e5b P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:18.641387 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:18.663125 4313 consensus_peers.cc:485] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:18.668004 4444 consensus_peers.cc:485] T 28da7347f982468281570ffa7503a854 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:18.691601 6771 consensus_peers.cc:485] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:18.694259 4575 consensus_peers.cc:485] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:18.822974 9125 raft_consensus.cc:1050] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P ed47c023452a497192f35d2f617bf9bd: attempting to promote NON_VOTER 923cfab926e7443f8ffd7b21ad0a9d91 to VOTER
I20240502 20:46:18.823393 9125 consensus_queue.cc:237] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } }
I20240502 20:46:18.824743 4638 raft_consensus.cc:1261] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEARNER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:18.824826 6834 raft_consensus.cc:1261] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:18.825258 12286 consensus_queue.cc:1035] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:46:18.825430 9125 consensus_queue.cc:1035] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 20:46:18.825523 4703 consensus_peers.cc:485] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:18.827205 12286 raft_consensus.cc:2932] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:18.827625 4638 raft_consensus.cc:2932] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:18.829437 6834 raft_consensus.cc:2932] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:18.830602 4228 catalog_manager.cc:5484] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: config changed from index 2 to 3, 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:18.920738 10776 raft_consensus.cc:1050] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833: attempting to promote NON_VOTER c3d26eafd51547acb780d4aac5ebe7cc to VOTER
I20240502 20:46:18.921193 10776 consensus_queue.cc:237] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } }
I20240502 20:46:18.922529 4507 raft_consensus.cc:1261] T b2508a5aa1c14b63971895953e149e17 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 LEARNER]: Refusing update from remote peer a4bf1df7fa6b4948be502c182bfbb833: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:18.922976 12293 consensus_queue.cc:1035] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 20:46:18.923336 6768 consensus_peers.cc:485] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:18.923336 6771 consensus_peers.cc:485] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:19.024840 12286 raft_consensus.cc:1050] T 3ba2f5d327f3403ea1604aae9485b960 P ed47c023452a497192f35d2f617bf9bd: attempting to promote NON_VOTER 923cfab926e7443f8ffd7b21ad0a9d91 to VOTER
I20240502 20:46:19.025301 12286 consensus_queue.cc:237] T 3ba2f5d327f3403ea1604aae9485b960 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } }
I20240502 20:46:19.026892 4374 raft_consensus.cc:1261] T 3ba2f5d327f3403ea1604aae9485b960 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:19.027109 4638 raft_consensus.cc:1261] T 3ba2f5d327f3403ea1604aae9485b960 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEARNER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 20:46:19.027693 11882 consensus_queue.cc:1035] T 3ba2f5d327f3403ea1604aae9485b960 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:46:19.027966 11882 consensus_queue.cc:1035] T 3ba2f5d327f3403ea1604aae9485b960 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 20:46:19.028448 4706 consensus_peers.cc:485] T 3ba2f5d327f3403ea1604aae9485b960 P ed47c023452a497192f35d2f617bf9bd -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:19.036015 12293 raft_consensus.cc:482] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7845c397230d49358ba2fad9db53f97c)
I20240502 20:46:19.036154 12293 raft_consensus.cc:504] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } }
I20240502 20:46:19.036542 12293 leader_election.cc:288] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097), 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
I20240502 20:46:19.036849 4507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75e2e48cd7a4497f9ccaae4f712f3ef0" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" is_pre_election: true
W20240502 20:46:19.037616 6770 leader_election.cc:334] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
I20240502 20:46:19.037683 6770 leader_election.cc:302] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4bf1df7fa6b4948be502c182bfbb833; no voters: 7845c397230d49358ba2fad9db53f97c, c3d26eafd51547acb780d4aac5ebe7cc
I20240502 20:46:19.037804 12293 raft_consensus.cc:2726] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 20:46:19.042081 9125 raft_consensus.cc:2932] T 3ba2f5d327f3403ea1604aae9485b960 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:19.042706 4374 raft_consensus.cc:2932] T 3ba2f5d327f3403ea1604aae9485b960 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:19.044016 4638 raft_consensus.cc:2932] T 3ba2f5d327f3403ea1604aae9485b960 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:19.045626 4231 catalog_manager.cc:5484] T 3ba2f5d327f3403ea1604aae9485b960 P ed47c023452a497192f35d2f617bf9bd reported cstate change: config changed from index 2 to 3, 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:19.048813 4768 consensus_queue.cc:237] T 3ba2f5d327f3403ea1604aae9485b960 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } }
I20240502 20:46:19.049919 4638 raft_consensus.cc:1261] T 3ba2f5d327f3403ea1604aae9485b960 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:19.050297 12286 consensus_queue.cc:1035] T 3ba2f5d327f3403ea1604aae9485b960 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:19.050372 4374 raft_consensus.cc:1261] T 3ba2f5d327f3403ea1604aae9485b960 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:19.050848 12286 consensus_queue.cc:1035] T 3ba2f5d327f3403ea1604aae9485b960 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:19.051878 9125 raft_consensus.cc:2932] T 3ba2f5d327f3403ea1604aae9485b960 P ed47c023452a497192f35d2f617bf9bd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:19.052412 4638 raft_consensus.cc:2932] T 3ba2f5d327f3403ea1604aae9485b960 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:19.053387 4222 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3ba2f5d327f3403ea1604aae9485b960 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:19.054085 4374 raft_consensus.cc:2932] T 3ba2f5d327f3403ea1604aae9485b960 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:19.054713 4228 catalog_manager.cc:5484] T 3ba2f5d327f3403ea1604aae9485b960 P ed47c023452a497192f35d2f617bf9bd reported cstate change: config changed from index 3 to 4, VOTER a47e5c56f13843018ac584914a8d4707 (127.30.32.71) evicted. New cstate: current_term: 1 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 20:46:19.090139 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
I20240502 20:46:19.097291 11608 raft_consensus.cc:482] T 6ed088a162a14d37b5da21ebb468a03b P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 201d2b6485084e7dad2cbb722af94779)
I20240502 20:46:19.097445 11608 raft_consensus.cc:504] T 6ed088a162a14d37b5da21ebb468a03b P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:19.097826 11608 leader_election.cc:288] T 6ed088a162a14d37b5da21ebb468a03b P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
I20240502 20:46:19.098261 6834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ed088a162a14d37b5da21ebb468a03b" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833" is_pre_election: true
W20240502 20:46:19.098943 4310 leader_election.cc:334] T 6ed088a162a14d37b5da21ebb468a03b P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
I20240502 20:46:19.099023 4310 leader_election.cc:302] T 6ed088a162a14d37b5da21ebb468a03b P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2; no voters: 201d2b6485084e7dad2cbb722af94779, a4bf1df7fa6b4948be502c182bfbb833
I20240502 20:46:19.099161 11608 raft_consensus.cc:2726] T 6ed088a162a14d37b5da21ebb468a03b P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:19.105512 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:19.115492 11605 raft_consensus.cc:482] T abe3bf6cb45a438fbd10bd5e78febc32 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 201d2b6485084e7dad2cbb722af94779)
I20240502 20:46:19.115600 11605 raft_consensus.cc:504] T abe3bf6cb45a438fbd10bd5e78febc32 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:19.115973 11605 leader_election.cc:288] T abe3bf6cb45a438fbd10bd5e78febc32 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643), ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:46:19.116384 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abe3bf6cb45a438fbd10bd5e78febc32" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd" is_pre_election: true
W20240502 20:46:19.117050 6768 leader_election.cc:334] T abe3bf6cb45a438fbd10bd5e78febc32 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
I20240502 20:46:19.117118 6768 leader_election.cc:302] T abe3bf6cb45a438fbd10bd5e78febc32 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4bf1df7fa6b4948be502c182bfbb833; no voters: 201d2b6485084e7dad2cbb722af94779, ed47c023452a497192f35d2f617bf9bd
I20240502 20:46:19.117228 11605 raft_consensus.cc:2726] T abe3bf6cb45a438fbd10bd5e78febc32 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 20:46:19.138218 12216 raft_consensus.cc:482] T abe3bf6cb45a438fbd10bd5e78febc32 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 201d2b6485084e7dad2cbb722af94779)
I20240502 20:46:19.138341 12216 raft_consensus.cc:504] T abe3bf6cb45a438fbd10bd5e78febc32 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:19.138813 12216 leader_election.cc:288] T abe3bf6cb45a438fbd10bd5e78febc32 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643), a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867)
I20240502 20:46:19.139535 6834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abe3bf6cb45a438fbd10bd5e78febc32" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833" is_pre_election: true
I20240502 20:46:19.139683 6834 raft_consensus.cc:2454] T abe3bf6cb45a438fbd10bd5e78febc32 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 1.
W20240502 20:46:19.139755 4703 leader_election.cc:334] T abe3bf6cb45a438fbd10bd5e78febc32 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
I20240502 20:46:19.139945 4704 leader_election.cc:302] T abe3bf6cb45a438fbd10bd5e78febc32 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a4bf1df7fa6b4948be502c182bfbb833, ed47c023452a497192f35d2f617bf9bd; no voters: 201d2b6485084e7dad2cbb722af94779
I20240502 20:46:19.140112 12216 raft_consensus.cc:2781] T abe3bf6cb45a438fbd10bd5e78febc32 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 20:46:19.140170 12216 raft_consensus.cc:482] T abe3bf6cb45a438fbd10bd5e78febc32 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 201d2b6485084e7dad2cbb722af94779)
I20240502 20:46:19.140213 12216 raft_consensus.cc:3037] T abe3bf6cb45a438fbd10bd5e78febc32 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:19.141206 12216 raft_consensus.cc:504] T abe3bf6cb45a438fbd10bd5e78febc32 P ed47c023452a497192f35d2f617bf9bd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:19.141496 12216 leader_election.cc:288] T abe3bf6cb45a438fbd10bd5e78febc32 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 2 election: Requested vote from peers 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643), a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867)
I20240502 20:46:19.141815 6834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abe3bf6cb45a438fbd10bd5e78febc32" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833"
I20240502 20:46:19.141933 6834 raft_consensus.cc:3037] T abe3bf6cb45a438fbd10bd5e78febc32 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Advancing to term 2
W20240502 20:46:19.142380 4703 leader_election.cc:334] T abe3bf6cb45a438fbd10bd5e78febc32 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
I20240502 20:46:19.142907 6834 raft_consensus.cc:2454] T abe3bf6cb45a438fbd10bd5e78febc32 P a4bf1df7fa6b4948be502c182bfbb833 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 2.
I20240502 20:46:19.143177 4704 leader_election.cc:302] T abe3bf6cb45a438fbd10bd5e78febc32 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a4bf1df7fa6b4948be502c182bfbb833, ed47c023452a497192f35d2f617bf9bd; no voters: 201d2b6485084e7dad2cbb722af94779
I20240502 20:46:19.143381 12216 raft_consensus.cc:2781] T abe3bf6cb45a438fbd10bd5e78febc32 P ed47c023452a497192f35d2f617bf9bd [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:46:19.143474 12216 raft_consensus.cc:686] T abe3bf6cb45a438fbd10bd5e78febc32 P ed47c023452a497192f35d2f617bf9bd [term 2 LEADER]: Becoming Leader. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Running, Role: LEADER
I20240502 20:46:19.143615 12216 consensus_queue.cc:237] T abe3bf6cb45a438fbd10bd5e78febc32 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } }
I20240502 20:46:19.145906 4231 catalog_manager.cc:5484] T abe3bf6cb45a438fbd10bd5e78febc32 P ed47c023452a497192f35d2f617bf9bd reported cstate change: term changed from 1 to 2, leader changed from 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) to ed47c023452a497192f35d2f617bf9bd (127.30.32.68). New cstate: current_term: 2 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } health_report { overall_health: HEALTHY } } }
I20240502 20:46:19.164129 11605 raft_consensus.cc:482] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7845c397230d49358ba2fad9db53f97c)
I20240502 20:46:19.164263 11605 raft_consensus.cc:504] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true } }
I20240502 20:46:19.164638 11605 leader_election.cc:288] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643), 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
W20240502 20:46:19.165551 6768 leader_election.cc:334] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:19.165674 6770 leader_election.cc:334] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
I20240502 20:46:19.165753 6770 leader_election.cc:302] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4bf1df7fa6b4948be502c182bfbb833; no voters: 201d2b6485084e7dad2cbb722af94779, 7845c397230d49358ba2fad9db53f97c
I20240502 20:46:19.165899 11605 raft_consensus.cc:2726] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:19.169194 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
I20240502 20:46:19.203648 11605 raft_consensus.cc:482] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 201d2b6485084e7dad2cbb722af94779)
I20240502 20:46:19.203773 11605 raft_consensus.cc:504] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } }
I20240502 20:46:19.204221 11605 leader_election.cc:288] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
W20240502 20:46:19.205291 6771 leader_election.cc:334] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:19.205520 6768 leader_election.cc:334] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
I20240502 20:46:19.205579 6768 leader_election.cc:302] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4bf1df7fa6b4948be502c182bfbb833; no voters: 201d2b6485084e7dad2cbb722af94779, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:19.205716 11605 raft_consensus.cc:2726] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 20:46:19.213457 11605 raft_consensus.cc:482] T ef4002126e8847809add182f262d62d9 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a47e5c56f13843018ac584914a8d4707)
I20240502 20:46:19.213598 11605 raft_consensus.cc:504] T ef4002126e8847809add182f262d62d9 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true } }
I20240502 20:46:19.213977 11605 leader_election.cc:288] T ef4002126e8847809add182f262d62d9 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:46:19.214537 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef4002126e8847809add182f262d62d9" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd" is_pre_election: true
W20240502 20:46:19.215271 6771 leader_election.cc:334] T ef4002126e8847809add182f262d62d9 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:19.215343 6771 leader_election.cc:302] T ef4002126e8847809add182f262d62d9 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4bf1df7fa6b4948be502c182bfbb833; no voters: a47e5c56f13843018ac584914a8d4707, ed47c023452a497192f35d2f617bf9bd
I20240502 20:46:19.215484 11605 raft_consensus.cc:2726] T ef4002126e8847809add182f262d62d9 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:19.223269 4222 catalog_manager.cc:4573] TS a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 3ba2f5d327f3403ea1604aae9485b960: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:19.235826 11608 raft_consensus.cc:482] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 201d2b6485084e7dad2cbb722af94779)
I20240502 20:46:19.235956 11608 raft_consensus.cc:504] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } }
I20240502 20:46:19.236344 11608 leader_election.cc:288] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
W20240502 20:46:19.237349 4310 leader_election.cc:334] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:19.237454 4313 leader_election.cc:334] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:19.237541 4313 leader_election.cc:302] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2; no voters: 201d2b6485084e7dad2cbb722af94779, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:19.237684 11608 raft_consensus.cc:2726] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 20:46:19.246203 12167 raft_consensus.cc:482] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a47e5c56f13843018ac584914a8d4707)
I20240502 20:46:19.246397 12167 raft_consensus.cc:504] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true } }
I20240502 20:46:19.246809 12167 leader_election.cc:288] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
I20240502 20:46:19.247395 11605 raft_consensus.cc:482] T 51232fccd98c461fb3d9016081f52ba5 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 201d2b6485084e7dad2cbb722af94779)
I20240502 20:46:19.247514 11605 raft_consensus.cc:504] T 51232fccd98c461fb3d9016081f52ba5 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } }
W20240502 20:46:19.247758 4575 leader_election.cc:334] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:19.247934 11605 leader_election.cc:288] T 51232fccd98c461fb3d9016081f52ba5 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643), 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
W20240502 20:46:19.248325 4574 leader_election.cc:334] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
I20240502 20:46:19.248396 4574 leader_election.cc:302] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91; no voters: 7845c397230d49358ba2fad9db53f97c, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:19.248649 4638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51232fccd98c461fb3d9016081f52ba5" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" is_pre_election: true
I20240502 20:46:19.248791 12279 raft_consensus.cc:2726] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:19.249229 6768 leader_election.cc:334] T 51232fccd98c461fb3d9016081f52ba5 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
I20240502 20:46:19.249296 6768 leader_election.cc:302] T 51232fccd98c461fb3d9016081f52ba5 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4bf1df7fa6b4948be502c182bfbb833; no voters: 201d2b6485084e7dad2cbb722af94779, 923cfab926e7443f8ffd7b21ad0a9d91
I20240502 20:46:19.249435 11605 raft_consensus.cc:2726] T 51232fccd98c461fb3d9016081f52ba5 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 20:46:19.262385 12279 raft_consensus.cc:482] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7845c397230d49358ba2fad9db53f97c)
I20240502 20:46:19.262528 12279 raft_consensus.cc:504] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } }
I20240502 20:46:19.262998 12279 leader_election.cc:288] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
I20240502 20:46:19.263830 12293 raft_consensus.cc:482] T 6ed088a162a14d37b5da21ebb468a03b P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 201d2b6485084e7dad2cbb722af94779)
I20240502 20:46:19.263963 12293 raft_consensus.cc:504] T 6ed088a162a14d37b5da21ebb468a03b P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
W20240502 20:46:19.264057 4574 leader_election.cc:334] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:19.264189 4572 leader_election.cc:334] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
I20240502 20:46:19.264290 4572 leader_election.cc:302] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91; no voters: 201d2b6485084e7dad2cbb722af94779, 7845c397230d49358ba2fad9db53f97c
I20240502 20:46:19.264413 12293 leader_election.cc:288] T 6ed088a162a14d37b5da21ebb468a03b P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
I20240502 20:46:19.264443 12279 raft_consensus.cc:2726] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 20:46:19.264973 4374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ed088a162a14d37b5da21ebb468a03b" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2" is_pre_election: true
I20240502 20:46:19.265163 4374 raft_consensus.cc:2454] T 6ed088a162a14d37b5da21ebb468a03b P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 1.
W20240502 20:46:19.265386 6768 leader_election.cc:334] T 6ed088a162a14d37b5da21ebb468a03b P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
I20240502 20:46:19.265492 6771 leader_election.cc:302] T 6ed088a162a14d37b5da21ebb468a03b P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a4bf1df7fa6b4948be502c182bfbb833, c92e6e46782849479c8a0c6d82276fb2; no voters: 201d2b6485084e7dad2cbb722af94779
I20240502 20:46:19.265659 12293 raft_consensus.cc:2781] T 6ed088a162a14d37b5da21ebb468a03b P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 20:46:19.265713 12293 raft_consensus.cc:482] T 6ed088a162a14d37b5da21ebb468a03b P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 201d2b6485084e7dad2cbb722af94779)
I20240502 20:46:19.265764 12293 raft_consensus.cc:3037] T 6ed088a162a14d37b5da21ebb468a03b P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:19.267024 12293 raft_consensus.cc:504] T 6ed088a162a14d37b5da21ebb468a03b P a4bf1df7fa6b4948be502c182bfbb833 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:19.267388 12293 leader_election.cc:288] T 6ed088a162a14d37b5da21ebb468a03b P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 election: Requested vote from peers c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
I20240502 20:46:19.267683 4374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ed088a162a14d37b5da21ebb468a03b" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2"
I20240502 20:46:19.267838 4374 raft_consensus.cc:3037] T 6ed088a162a14d37b5da21ebb468a03b P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Advancing to term 2
W20240502 20:46:19.268218 6768 leader_election.cc:334] T 6ed088a162a14d37b5da21ebb468a03b P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
I20240502 20:46:19.268847 4374 raft_consensus.cc:2454] T 6ed088a162a14d37b5da21ebb468a03b P c92e6e46782849479c8a0c6d82276fb2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4bf1df7fa6b4948be502c182bfbb833 in term 2.
I20240502 20:46:19.269131 6771 leader_election.cc:302] T 6ed088a162a14d37b5da21ebb468a03b P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a4bf1df7fa6b4948be502c182bfbb833, c92e6e46782849479c8a0c6d82276fb2; no voters: 201d2b6485084e7dad2cbb722af94779
I20240502 20:46:19.269351 12293 raft_consensus.cc:2781] T 6ed088a162a14d37b5da21ebb468a03b P a4bf1df7fa6b4948be502c182bfbb833 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:46:19.269435 12293 raft_consensus.cc:686] T 6ed088a162a14d37b5da21ebb468a03b P a4bf1df7fa6b4948be502c182bfbb833 [term 2 LEADER]: Becoming Leader. State: Replica: a4bf1df7fa6b4948be502c182bfbb833, State: Running, Role: LEADER
I20240502 20:46:19.269573 12293 consensus_queue.cc:237] T 6ed088a162a14d37b5da21ebb468a03b P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:19.271472 4231 catalog_manager.cc:5484] T 6ed088a162a14d37b5da21ebb468a03b P a4bf1df7fa6b4948be502c182bfbb833 reported cstate change: term changed from 1 to 2, leader changed from 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) to a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70). New cstate: current_term: 2 leader_uuid: "a4bf1df7fa6b4948be502c182bfbb833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:19.297340 12279 raft_consensus.cc:482] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7845c397230d49358ba2fad9db53f97c)
I20240502 20:46:19.297456 12279 raft_consensus.cc:504] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: true } }
I20240502 20:46:19.297873 12279 leader_election.cc:288] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
W20240502 20:46:19.298784 4574 leader_election.cc:334] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:19.298879 4572 leader_election.cc:334] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
I20240502 20:46:19.298976 4572 leader_election.cc:302] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91; no voters: 201d2b6485084e7dad2cbb722af94779, 7845c397230d49358ba2fad9db53f97c
I20240502 20:46:19.299105 12279 raft_consensus.cc:2726] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 20:46:19.300315 12279 raft_consensus.cc:482] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7845c397230d49358ba2fad9db53f97c)
I20240502 20:46:19.300417 12279 raft_consensus.cc:504] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:19.300740 12279 leader_election.cc:288] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643), 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
W20240502 20:46:19.301609 4572 leader_election.cc:334] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:19.301694 4574 leader_election.cc:334] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
I20240502 20:46:19.301767 4574 leader_election.cc:302] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91; no voters: 201d2b6485084e7dad2cbb722af94779, 7845c397230d49358ba2fad9db53f97c
I20240502 20:46:19.301887 12279 raft_consensus.cc:2726] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 20:46:19.304953 12279 raft_consensus.cc:482] T 51232fccd98c461fb3d9016081f52ba5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 201d2b6485084e7dad2cbb722af94779)
I20240502 20:46:19.305091 12279 raft_consensus.cc:504] T 51232fccd98c461fb3d9016081f52ba5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } }
I20240502 20:46:19.305441 12279 leader_election.cc:288] T 51232fccd98c461fb3d9016081f52ba5 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643), a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867)
I20240502 20:46:19.305876 6834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51232fccd98c461fb3d9016081f52ba5" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833" is_pre_election: true
I20240502 20:46:19.306041 6834 raft_consensus.cc:2454] T 51232fccd98c461fb3d9016081f52ba5 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 1.
I20240502 20:46:19.306351 4573 leader_election.cc:302] T 51232fccd98c461fb3d9016081f52ba5 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, a4bf1df7fa6b4948be502c182bfbb833; no voters:
W20240502 20:46:19.306459 4572 leader_election.cc:334] T 51232fccd98c461fb3d9016081f52ba5 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
I20240502 20:46:19.306515 12279 raft_consensus.cc:2781] T 51232fccd98c461fb3d9016081f52ba5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 20:46:19.306571 12279 raft_consensus.cc:482] T 51232fccd98c461fb3d9016081f52ba5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 201d2b6485084e7dad2cbb722af94779)
I20240502 20:46:19.306625 12279 raft_consensus.cc:3037] T 51232fccd98c461fb3d9016081f52ba5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:19.307536 12279 raft_consensus.cc:504] T 51232fccd98c461fb3d9016081f52ba5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } }
I20240502 20:46:19.307880 12279 leader_election.cc:288] T 51232fccd98c461fb3d9016081f52ba5 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 election: Requested vote from peers 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643), a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867)
I20240502 20:46:19.308259 6834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51232fccd98c461fb3d9016081f52ba5" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833"
I20240502 20:46:19.308399 6834 raft_consensus.cc:3037] T 51232fccd98c461fb3d9016081f52ba5 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Advancing to term 2
W20240502 20:46:19.308738 4572 leader_election.cc:334] T 51232fccd98c461fb3d9016081f52ba5 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
I20240502 20:46:19.309489 6834 raft_consensus.cc:2454] T 51232fccd98c461fb3d9016081f52ba5 P a4bf1df7fa6b4948be502c182bfbb833 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 2.
I20240502 20:46:19.309784 4573 leader_election.cc:302] T 51232fccd98c461fb3d9016081f52ba5 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, a4bf1df7fa6b4948be502c182bfbb833; no voters: 201d2b6485084e7dad2cbb722af94779
I20240502 20:46:19.309962 12279 raft_consensus.cc:2781] T 51232fccd98c461fb3d9016081f52ba5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:46:19.310081 12279 raft_consensus.cc:686] T 51232fccd98c461fb3d9016081f52ba5 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 2 LEADER]: Becoming Leader. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Running, Role: LEADER
I20240502 20:46:19.310226 12279 consensus_queue.cc:237] T 51232fccd98c461fb3d9016081f52ba5 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } }
I20240502 20:46:19.312094 4231 catalog_manager.cc:5484] T 51232fccd98c461fb3d9016081f52ba5 P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: term changed from 1 to 2, leader changed from 201d2b6485084e7dad2cbb722af94779 (127.30.32.72) to 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67). New cstate: current_term: 2 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:19.364676 12278 raft_consensus.cc:482] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7845c397230d49358ba2fad9db53f97c)
I20240502 20:46:19.364845 12278 raft_consensus.cc:504] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: false } }
I20240502 20:46:19.365259 12278 leader_election.cc:288] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
W20240502 20:46:19.366297 4312 leader_election.cc:334] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:19.366422 4310 leader_election.cc:334] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
I20240502 20:46:19.366495 4310 leader_election.cc:302] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2; no voters: 201d2b6485084e7dad2cbb722af94779, 7845c397230d49358ba2fad9db53f97c
I20240502 20:46:19.366631 12278 raft_consensus.cc:2726] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 20:46:19.377727 12278 raft_consensus.cc:482] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a47e5c56f13843018ac584914a8d4707)
I20240502 20:46:19.377839 12278 raft_consensus.cc:504] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:19.378208 12278 leader_election.cc:288] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
W20240502 20:46:19.379104 4312 leader_election.cc:334] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:19.379204 4313 leader_election.cc:334] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:19.379299 4313 leader_election.cc:302] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2; no voters: 7845c397230d49358ba2fad9db53f97c, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:19.379427 12278 raft_consensus.cc:2726] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 20:46:19.380206 11600 raft_consensus.cc:482] T 77dc51e6a85a4ddcb64db12b40c5810f P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7845c397230d49358ba2fad9db53f97c)
I20240502 20:46:19.380355 11600 raft_consensus.cc:504] T 77dc51e6a85a4ddcb64db12b40c5810f P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:19.380748 11600 leader_election.cc:288] T 77dc51e6a85a4ddcb64db12b40c5810f P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67:42141)
I20240502 20:46:19.381112 4638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77dc51e6a85a4ddcb64db12b40c5810f" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" is_pre_election: true
W20240502 20:46:19.381685 4443 leader_election.cc:334] T 77dc51e6a85a4ddcb64db12b40c5810f P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
I20240502 20:46:19.381752 4443 leader_election.cc:302] T 77dc51e6a85a4ddcb64db12b40c5810f P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3d26eafd51547acb780d4aac5ebe7cc; no voters: 7845c397230d49358ba2fad9db53f97c, 923cfab926e7443f8ffd7b21ad0a9d91
I20240502 20:46:19.381912 11600 raft_consensus.cc:2726] T 77dc51e6a85a4ddcb64db12b40c5810f P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 20:46:19.386811 12278 raft_consensus.cc:482] T 9258667ccc3e4ef8b26fe75531dd4591 P c92e6e46782849479c8a0c6d82276fb2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a47e5c56f13843018ac584914a8d4707)
I20240502 20:46:19.386916 12278 raft_consensus.cc:504] T 9258667ccc3e4ef8b26fe75531dd4591 P c92e6e46782849479c8a0c6d82276fb2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } }
I20240502 20:46:19.387281 12278 leader_election.cc:288] T 9258667ccc3e4ef8b26fe75531dd4591 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), ed47c023452a497192f35d2f617bf9bd (127.30.32.68:38193)
I20240502 20:46:19.387856 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9258667ccc3e4ef8b26fe75531dd4591" candidate_uuid: "c92e6e46782849479c8a0c6d82276fb2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ed47c023452a497192f35d2f617bf9bd" is_pre_election: true
W20240502 20:46:19.388236 4313 leader_election.cc:334] T 9258667ccc3e4ef8b26fe75531dd4591 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:19.388303 4313 leader_election.cc:302] T 9258667ccc3e4ef8b26fe75531dd4591 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2; no voters: a47e5c56f13843018ac584914a8d4707, ed47c023452a497192f35d2f617bf9bd
I20240502 20:46:19.388417 12278 raft_consensus.cc:2726] T 9258667ccc3e4ef8b26fe75531dd4591 P c92e6e46782849479c8a0c6d82276fb2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 20:46:19.407146 12286 raft_consensus.cc:482] T 9258667ccc3e4ef8b26fe75531dd4591 P ed47c023452a497192f35d2f617bf9bd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a47e5c56f13843018ac584914a8d4707)
I20240502 20:46:19.407275 12286 raft_consensus.cc:504] T 9258667ccc3e4ef8b26fe75531dd4591 P ed47c023452a497192f35d2f617bf9bd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } }
I20240502 20:46:19.407670 12286 leader_election.cc:288] T 9258667ccc3e4ef8b26fe75531dd4591 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
I20240502 20:46:19.408082 4374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9258667ccc3e4ef8b26fe75531dd4591" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2" is_pre_election: true
I20240502 20:46:19.408236 4374 raft_consensus.cc:2454] T 9258667ccc3e4ef8b26fe75531dd4591 P c92e6e46782849479c8a0c6d82276fb2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 2.
I20240502 20:46:19.408694 4706 leader_election.cc:302] T 9258667ccc3e4ef8b26fe75531dd4591 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2, ed47c023452a497192f35d2f617bf9bd; no voters:
W20240502 20:46:19.408829 4706 leader_election.cc:334] T 9258667ccc3e4ef8b26fe75531dd4591 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:19.409054 12286 raft_consensus.cc:2781] T 9258667ccc3e4ef8b26fe75531dd4591 P ed47c023452a497192f35d2f617bf9bd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 20:46:19.409116 12286 raft_consensus.cc:482] T 9258667ccc3e4ef8b26fe75531dd4591 P ed47c023452a497192f35d2f617bf9bd [term 2 FOLLOWER]: Starting leader election (detected failure of leader a47e5c56f13843018ac584914a8d4707)
I20240502 20:46:19.409170 12286 raft_consensus.cc:3037] T 9258667ccc3e4ef8b26fe75531dd4591 P ed47c023452a497192f35d2f617bf9bd [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:46:19.410010 12286 raft_consensus.cc:504] T 9258667ccc3e4ef8b26fe75531dd4591 P ed47c023452a497192f35d2f617bf9bd [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } }
I20240502 20:46:19.410310 12286 leader_election.cc:288] T 9258667ccc3e4ef8b26fe75531dd4591 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 3 election: Requested vote from peers c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65:33827), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
I20240502 20:46:19.411181 4374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9258667ccc3e4ef8b26fe75531dd4591" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c92e6e46782849479c8a0c6d82276fb2"
W20240502 20:46:19.411290 4706 leader_election.cc:334] T 9258667ccc3e4ef8b26fe75531dd4591 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:19.411306 4374 raft_consensus.cc:3037] T 9258667ccc3e4ef8b26fe75531dd4591 P c92e6e46782849479c8a0c6d82276fb2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:46:19.412108 4374 raft_consensus.cc:2454] T 9258667ccc3e4ef8b26fe75531dd4591 P c92e6e46782849479c8a0c6d82276fb2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 3.
I20240502 20:46:19.412355 4706 leader_election.cc:302] T 9258667ccc3e4ef8b26fe75531dd4591 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2, ed47c023452a497192f35d2f617bf9bd; no voters: a47e5c56f13843018ac584914a8d4707
I20240502 20:46:19.412497 12286 raft_consensus.cc:2781] T 9258667ccc3e4ef8b26fe75531dd4591 P ed47c023452a497192f35d2f617bf9bd [term 3 FOLLOWER]: Leader election won for term 3
I20240502 20:46:19.412592 12286 raft_consensus.cc:686] T 9258667ccc3e4ef8b26fe75531dd4591 P ed47c023452a497192f35d2f617bf9bd [term 3 LEADER]: Becoming Leader. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Running, Role: LEADER
I20240502 20:46:19.412747 12286 consensus_queue.cc:237] T 9258667ccc3e4ef8b26fe75531dd4591 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } }
I20240502 20:46:19.414834 4231 catalog_manager.cc:5484] T 9258667ccc3e4ef8b26fe75531dd4591 P ed47c023452a497192f35d2f617bf9bd reported cstate change: term changed from 2 to 3, leader changed from a47e5c56f13843018ac584914a8d4707 (127.30.32.71) to ed47c023452a497192f35d2f617bf9bd (127.30.32.68). New cstate: current_term: 3 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:19.437036 12293 raft_consensus.cc:482] T 747e7331d7ad4dd4b132672254037e1b P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a47e5c56f13843018ac584914a8d4707)
I20240502 20:46:19.437167 12293 raft_consensus.cc:504] T 747e7331d7ad4dd4b132672254037e1b P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } }
I20240502 20:46:19.437536 12293 leader_election.cc:288] T 747e7331d7ad4dd4b132672254037e1b P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097)
I20240502 20:46:19.437978 4507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "747e7331d7ad4dd4b132672254037e1b" candidate_uuid: "a4bf1df7fa6b4948be502c182bfbb833" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" is_pre_election: true
W20240502 20:46:19.438424 6771 leader_election.cc:334] T 747e7331d7ad4dd4b132672254037e1b P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:19.438508 6770 leader_election.cc:302] T 747e7331d7ad4dd4b132672254037e1b P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4bf1df7fa6b4948be502c182bfbb833; no voters: a47e5c56f13843018ac584914a8d4707, c3d26eafd51547acb780d4aac5ebe7cc
I20240502 20:46:19.438658 12293 raft_consensus.cc:2726] T 747e7331d7ad4dd4b132672254037e1b P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 20:46:19.441557 11600 raft_consensus.cc:482] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a47e5c56f13843018ac584914a8d4707)
I20240502 20:46:19.441651 11600 raft_consensus.cc:504] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:19.442001 11600 leader_election.cc:288] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
W20240502 20:46:19.442973 4444 leader_election.cc:334] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:19.443061 4441 leader_election.cc:334] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
I20240502 20:46:19.443140 4441 leader_election.cc:302] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3d26eafd51547acb780d4aac5ebe7cc; no voters: 201d2b6485084e7dad2cbb722af94779, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:19.443291 11600 raft_consensus.cc:2726] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 20:46:19.458362 12286 raft_consensus.cc:482] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a47e5c56f13843018ac584914a8d4707)
I20240502 20:46:19.458492 12286 raft_consensus.cc:504] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } }
I20240502 20:46:19.458948 12286 leader_election.cc:288] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
W20240502 20:46:19.459861 4705 leader_election.cc:334] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:19.459945 4706 leader_election.cc:334] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:19.460024 4706 leader_election.cc:302] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed47c023452a497192f35d2f617bf9bd; no voters: 7845c397230d49358ba2fad9db53f97c, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:19.460155 12286 raft_consensus.cc:2726] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 20:46:19.490713 12278 raft_consensus.cc:482] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7845c397230d49358ba2fad9db53f97c)
I20240502 20:46:19.490871 12278 raft_consensus.cc:504] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:19.491302 12278 leader_election.cc:288] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
W20240502 20:46:19.492321 4312 leader_election.cc:334] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:19.492417 4313 leader_election.cc:334] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:19.492501 4313 leader_election.cc:302] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2; no voters: 7845c397230d49358ba2fad9db53f97c, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:19.492651 12278 raft_consensus.cc:2726] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 20:46:19.504103 11600 raft_consensus.cc:482] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7845c397230d49358ba2fad9db53f97c)
I20240502 20:46:19.504240 11600 raft_consensus.cc:504] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } }
I20240502 20:46:19.504623 11600 leader_election.cc:288] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867), 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
I20240502 20:46:19.505019 6834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75e2e48cd7a4497f9ccaae4f712f3ef0" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833" is_pre_election: true
I20240502 20:46:19.505180 6834 raft_consensus.cc:2454] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 1.
I20240502 20:46:19.505492 4442 leader_election.cc:302] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4bf1df7fa6b4948be502c182bfbb833, c3d26eafd51547acb780d4aac5ebe7cc; no voters:
I20240502 20:46:19.505673 11600 raft_consensus.cc:2781] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 20:46:19.505726 11600 raft_consensus.cc:482] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7845c397230d49358ba2fad9db53f97c)
I20240502 20:46:19.505777 11600 raft_consensus.cc:3037] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Advancing to term 2
W20240502 20:46:19.506006 4443 leader_election.cc:334] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
I20240502 20:46:19.506793 11600 raft_consensus.cc:504] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P c3d26eafd51547acb780d4aac5ebe7cc [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } }
I20240502 20:46:19.507107 11600 leader_election.cc:288] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 2 election: Requested vote from peers a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867), 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
I20240502 20:46:19.507449 6834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75e2e48cd7a4497f9ccaae4f712f3ef0" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833"
I20240502 20:46:19.507589 6834 raft_consensus.cc:3037] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Advancing to term 2
W20240502 20:46:19.507987 4443 leader_election.cc:334] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
I20240502 20:46:19.508533 6834 raft_consensus.cc:2454] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P a4bf1df7fa6b4948be502c182bfbb833 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 2.
I20240502 20:46:19.508827 4442 leader_election.cc:302] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a4bf1df7fa6b4948be502c182bfbb833, c3d26eafd51547acb780d4aac5ebe7cc; no voters: 7845c397230d49358ba2fad9db53f97c
I20240502 20:46:19.509021 11600 raft_consensus.cc:2781] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P c3d26eafd51547acb780d4aac5ebe7cc [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:46:19.509107 11600 raft_consensus.cc:686] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P c3d26eafd51547acb780d4aac5ebe7cc [term 2 LEADER]: Becoming Leader. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Running, Role: LEADER
I20240502 20:46:19.509258 11600 consensus_queue.cc:237] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } }
I20240502 20:46:19.511489 4231 catalog_manager.cc:5484] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: term changed from 1 to 2, leader changed from 7845c397230d49358ba2fad9db53f97c (127.30.32.69) to c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66). New cstate: current_term: 2 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:19.526777 12278 raft_consensus.cc:482] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a47e5c56f13843018ac584914a8d4707)
I20240502 20:46:19.526907 12278 raft_consensus.cc:504] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:19.527338 12278 leader_election.cc:288] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
W20240502 20:46:19.528244 4310 leader_election.cc:334] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:19.528409 4313 leader_election.cc:334] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:19.528493 4313 leader_election.cc:302] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2; no voters: 201d2b6485084e7dad2cbb722af94779, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:19.528609 12278 raft_consensus.cc:2726] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:19.574810 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:19.576825 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:19.580330 12167 raft_consensus.cc:482] T 77dc51e6a85a4ddcb64db12b40c5810f P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7845c397230d49358ba2fad9db53f97c)
I20240502 20:46:19.580451 12167 raft_consensus.cc:504] T 77dc51e6a85a4ddcb64db12b40c5810f P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:19.580844 12167 leader_election.cc:288] T 77dc51e6a85a4ddcb64db12b40c5810f P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097)
W20240502 20:46:19.580961 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
I20240502 20:46:19.581558 4507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77dc51e6a85a4ddcb64db12b40c5810f" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" is_pre_election: true
I20240502 20:46:19.581712 4507 raft_consensus.cc:2454] T 77dc51e6a85a4ddcb64db12b40c5810f P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 1.
W20240502 20:46:19.581835 4574 leader_election.cc:334] T 77dc51e6a85a4ddcb64db12b40c5810f P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
I20240502 20:46:19.581977 4574 leader_election.cc:302] T 77dc51e6a85a4ddcb64db12b40c5810f P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, c3d26eafd51547acb780d4aac5ebe7cc; no voters: 7845c397230d49358ba2fad9db53f97c
I20240502 20:46:19.582125 12167 raft_consensus.cc:2781] T 77dc51e6a85a4ddcb64db12b40c5810f P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 20:46:19.582194 12167 raft_consensus.cc:482] T 77dc51e6a85a4ddcb64db12b40c5810f P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7845c397230d49358ba2fad9db53f97c)
I20240502 20:46:19.582255 12167 raft_consensus.cc:3037] T 77dc51e6a85a4ddcb64db12b40c5810f P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:19.583171 12167 raft_consensus.cc:504] T 77dc51e6a85a4ddcb64db12b40c5810f P 923cfab926e7443f8ffd7b21ad0a9d91 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:19.583493 12167 leader_election.cc:288] T 77dc51e6a85a4ddcb64db12b40c5810f P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 election: Requested vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66:42097)
I20240502 20:46:19.584064 4507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77dc51e6a85a4ddcb64db12b40c5810f" candidate_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c3d26eafd51547acb780d4aac5ebe7cc"
I20240502 20:46:19.584189 4507 raft_consensus.cc:3037] T 77dc51e6a85a4ddcb64db12b40c5810f P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Advancing to term 2
W20240502 20:46:19.584707 4574 leader_election.cc:334] T 77dc51e6a85a4ddcb64db12b40c5810f P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
I20240502 20:46:19.585398 4507 raft_consensus.cc:2454] T 77dc51e6a85a4ddcb64db12b40c5810f P c3d26eafd51547acb780d4aac5ebe7cc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 923cfab926e7443f8ffd7b21ad0a9d91 in term 2.
I20240502 20:46:19.585708 4574 leader_election.cc:302] T 77dc51e6a85a4ddcb64db12b40c5810f P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91, c3d26eafd51547acb780d4aac5ebe7cc; no voters: 7845c397230d49358ba2fad9db53f97c
I20240502 20:46:19.585881 12167 raft_consensus.cc:2781] T 77dc51e6a85a4ddcb64db12b40c5810f P 923cfab926e7443f8ffd7b21ad0a9d91 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:46:19.585978 12167 raft_consensus.cc:686] T 77dc51e6a85a4ddcb64db12b40c5810f P 923cfab926e7443f8ffd7b21ad0a9d91 [term 2 LEADER]: Becoming Leader. State: Replica: 923cfab926e7443f8ffd7b21ad0a9d91, State: Running, Role: LEADER
I20240502 20:46:19.586094 12167 consensus_queue.cc:237] T 77dc51e6a85a4ddcb64db12b40c5810f P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } }
I20240502 20:46:19.588357 4231 catalog_manager.cc:5484] T 77dc51e6a85a4ddcb64db12b40c5810f P 923cfab926e7443f8ffd7b21ad0a9d91 reported cstate change: term changed from 1 to 2, leader changed from 7845c397230d49358ba2fad9db53f97c (127.30.32.69) to 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67). New cstate: current_term: 2 leader_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:19.602726 12286 raft_consensus.cc:482] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a47e5c56f13843018ac584914a8d4707)
I20240502 20:46:19.602874 12286 raft_consensus.cc:504] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true } }
I20240502 20:46:19.603483 12286 leader_election.cc:288] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867)
I20240502 20:46:19.603966 6834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef4002126e8847809add182f262d62d9" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833" is_pre_election: true
I20240502 20:46:19.604122 6834 raft_consensus.cc:2454] T ef4002126e8847809add182f262d62d9 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 1.
I20240502 20:46:19.604439 4704 leader_election.cc:302] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4bf1df7fa6b4948be502c182bfbb833, ed47c023452a497192f35d2f617bf9bd; no voters:
I20240502 20:46:19.604650 12216 raft_consensus.cc:2781] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 20:46:19.604713 12216 raft_consensus.cc:482] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Starting leader election (detected failure of leader a47e5c56f13843018ac584914a8d4707)
I20240502 20:46:19.604754 12216 raft_consensus.cc:3037] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:19.605798 12216 raft_consensus.cc:504] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true } }
I20240502 20:46:19.606129 12216 leader_election.cc:288] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 2 election: Requested vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867)
W20240502 20:46:19.606503 4706 leader_election.cc:334] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:19.607045 4703 consensus_peers.cc:485] T abe3bf6cb45a438fbd10bd5e78febc32 P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:19.607187 4706 leader_election.cc:334] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:19.609447 6834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef4002126e8847809add182f262d62d9" candidate_uuid: "ed47c023452a497192f35d2f617bf9bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833"
I20240502 20:46:19.609591 6834 raft_consensus.cc:3037] T ef4002126e8847809add182f262d62d9 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:19.610442 6834 raft_consensus.cc:2454] T ef4002126e8847809add182f262d62d9 P a4bf1df7fa6b4948be502c182bfbb833 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed47c023452a497192f35d2f617bf9bd in term 2.
I20240502 20:46:19.610756 4704 leader_election.cc:302] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a4bf1df7fa6b4948be502c182bfbb833, ed47c023452a497192f35d2f617bf9bd; no voters: a47e5c56f13843018ac584914a8d4707
I20240502 20:46:19.611224 12286 raft_consensus.cc:2781] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:46:19.611392 12286 raft_consensus.cc:686] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd [term 2 LEADER]: Becoming Leader. State: Replica: ed47c023452a497192f35d2f617bf9bd, State: Running, Role: LEADER
I20240502 20:46:19.611557 12286 consensus_queue.cc:237] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true } }
I20240502 20:46:19.613689 4231 catalog_manager.cc:5484] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd reported cstate change: term changed from 1 to 2, leader changed from a47e5c56f13843018ac584914a8d4707 (127.30.32.71) to ed47c023452a497192f35d2f617bf9bd (127.30.32.68). New cstate: current_term: 2 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:19.618973 11605 raft_consensus.cc:482] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a47e5c56f13843018ac584914a8d4707)
I20240502 20:46:19.619102 11605 raft_consensus.cc:504] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } }
I20240502 20:46:19.619485 11605 leader_election.cc:288] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
W20240502 20:46:19.620540 6771 leader_election.cc:334] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:19.620667 6770 leader_election.cc:334] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
I20240502 20:46:19.620733 6770 leader_election.cc:302] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4bf1df7fa6b4948be502c182bfbb833; no voters: 7845c397230d49358ba2fad9db53f97c, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:19.620887 6834 raft_consensus.cc:1261] T abe3bf6cb45a438fbd10bd5e78febc32 P a4bf1df7fa6b4948be502c182bfbb833 [term 2 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 20:46:19.620942 12293 raft_consensus.cc:2726] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 20:46:19.621246 12286 consensus_queue.cc:1035] T abe3bf6cb45a438fbd10bd5e78febc32 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:46:19.643563 11600 raft_consensus.cc:482] T 747e7331d7ad4dd4b132672254037e1b P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a47e5c56f13843018ac584914a8d4707)
I20240502 20:46:19.643685 11600 raft_consensus.cc:504] T 747e7331d7ad4dd4b132672254037e1b P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } }
I20240502 20:46:19.644101 11600 leader_election.cc:288] T 747e7331d7ad4dd4b132672254037e1b P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
I20240502 20:46:19.644469 6834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "747e7331d7ad4dd4b132672254037e1b" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833" is_pre_election: true
I20240502 20:46:19.644601 6834 raft_consensus.cc:2454] T 747e7331d7ad4dd4b132672254037e1b P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 1.
I20240502 20:46:19.644898 4442 leader_election.cc:302] T 747e7331d7ad4dd4b132672254037e1b P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4bf1df7fa6b4948be502c182bfbb833, c3d26eafd51547acb780d4aac5ebe7cc; no voters:
I20240502 20:46:19.645094 11600 raft_consensus.cc:2781] T 747e7331d7ad4dd4b132672254037e1b P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 20:46:19.645174 11600 raft_consensus.cc:482] T 747e7331d7ad4dd4b132672254037e1b P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Starting leader election (detected failure of leader a47e5c56f13843018ac584914a8d4707)
W20240502 20:46:19.645185 4444 leader_election.cc:334] T 747e7331d7ad4dd4b132672254037e1b P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:19.645233 11600 raft_consensus.cc:3037] T 747e7331d7ad4dd4b132672254037e1b P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:19.646554 11600 raft_consensus.cc:504] T 747e7331d7ad4dd4b132672254037e1b P c3d26eafd51547acb780d4aac5ebe7cc [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } }
I20240502 20:46:19.647027 11600 leader_election.cc:288] T 747e7331d7ad4dd4b132672254037e1b P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 2 election: Requested vote from peers a4bf1df7fa6b4948be502c182bfbb833 (127.30.32.70:41867), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
I20240502 20:46:19.647363 6834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "747e7331d7ad4dd4b132672254037e1b" candidate_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4bf1df7fa6b4948be502c182bfbb833"
I20240502 20:46:19.647500 6834 raft_consensus.cc:3037] T 747e7331d7ad4dd4b132672254037e1b P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Advancing to term 2
W20240502 20:46:19.647935 4444 leader_election.cc:334] T 747e7331d7ad4dd4b132672254037e1b P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:19.648574 6834 raft_consensus.cc:2454] T 747e7331d7ad4dd4b132672254037e1b P a4bf1df7fa6b4948be502c182bfbb833 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3d26eafd51547acb780d4aac5ebe7cc in term 2.
I20240502 20:46:19.648941 4442 leader_election.cc:302] T 747e7331d7ad4dd4b132672254037e1b P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a4bf1df7fa6b4948be502c182bfbb833, c3d26eafd51547acb780d4aac5ebe7cc; no voters: a47e5c56f13843018ac584914a8d4707
I20240502 20:46:19.649150 11600 raft_consensus.cc:2781] T 747e7331d7ad4dd4b132672254037e1b P c3d26eafd51547acb780d4aac5ebe7cc [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:46:19.649245 11600 raft_consensus.cc:686] T 747e7331d7ad4dd4b132672254037e1b P c3d26eafd51547acb780d4aac5ebe7cc [term 2 LEADER]: Becoming Leader. State: Replica: c3d26eafd51547acb780d4aac5ebe7cc, State: Running, Role: LEADER
I20240502 20:46:19.649394 11600 consensus_queue.cc:237] T 747e7331d7ad4dd4b132672254037e1b P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } }
I20240502 20:46:19.651477 4230 catalog_manager.cc:5484] T 747e7331d7ad4dd4b132672254037e1b P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: term changed from 1 to 2, leader changed from a47e5c56f13843018ac584914a8d4707 (127.30.32.71) to c3d26eafd51547acb780d4aac5ebe7cc (127.30.32.66). New cstate: current_term: 2 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:19.694517 4377 raft_consensus.cc:1261] T 6ed088a162a14d37b5da21ebb468a03b P c92e6e46782849479c8a0c6d82276fb2 [term 2 FOLLOWER]: Refusing update from remote peer a4bf1df7fa6b4948be502c182bfbb833: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 20:46:19.694931 11605 consensus_queue.cc:1035] T 6ed088a162a14d37b5da21ebb468a03b P a4bf1df7fa6b4948be502c182bfbb833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 20:46:19.698443 6768 consensus_peers.cc:485] T 6ed088a162a14d37b5da21ebb468a03b P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:19.754654 6834 raft_consensus.cc:1261] T 51232fccd98c461fb3d9016081f52ba5 P a4bf1df7fa6b4948be502c182bfbb833 [term 2 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 20:46:19.755100 12167 consensus_queue.cc:1035] T 51232fccd98c461fb3d9016081f52ba5 P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 20:46:19.758531 4572 consensus_peers.cc:485] T 51232fccd98c461fb3d9016081f52ba5 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:19.937855 6834 raft_consensus.cc:1261] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P a4bf1df7fa6b4948be502c182bfbb833 [term 2 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 20:46:19.938258 11600 consensus_queue.cc:1035] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:46:19.940835 4638 raft_consensus.cc:3037] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 LEARNER]: Advancing to term 2
W20240502 20:46:19.941406 4443 consensus_peers.cc:485] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:19.942050 4638 raft_consensus.cc:1261] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 2 LEARNER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 20:46:19.942545 12321 consensus_queue.cc:1035] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: NON_VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 20:46:19.945375 12321 raft_consensus.cc:1050] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P c3d26eafd51547acb780d4aac5ebe7cc: attempting to promote NON_VOTER 923cfab926e7443f8ffd7b21ad0a9d91 to VOTER
I20240502 20:46:19.945832 12321 consensus_queue.cc:237] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } }
I20240502 20:46:19.947067 6834 raft_consensus.cc:1261] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P a4bf1df7fa6b4948be502c182bfbb833 [term 2 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 20:46:19.947137 4638 raft_consensus.cc:1261] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 2 LEARNER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 20:46:19.947504 11600 consensus_queue.cc:1035] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:19.947714 12320 consensus_queue.cc:1035] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 20:46:19.949036 4443 consensus_peers.cc:485] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:19.950330 11600 raft_consensus.cc:2932] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P c3d26eafd51547acb780d4aac5ebe7cc [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:19.950775 4638 raft_consensus.cc:2932] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:19.952716 6834 raft_consensus.cc:2932] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P a4bf1df7fa6b4948be502c182bfbb833 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } }
I20240502 20:46:19.953202 4231 catalog_manager.cc:5484] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P c3d26eafd51547acb780d4aac5ebe7cc reported cstate change: config changed from index 2 to 4, 923cfab926e7443f8ffd7b21ad0a9d91 (127.30.32.67) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 20:46:19.969940 4706 consensus_peers.cc:485] T 9258667ccc3e4ef8b26fe75531dd4591 P ed47c023452a497192f35d2f617bf9bd -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:19.975124 4377 raft_consensus.cc:1261] T 9258667ccc3e4ef8b26fe75531dd4591 P c92e6e46782849479c8a0c6d82276fb2 [term 3 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240502 20:46:19.975544 12286 consensus_queue.cc:1035] T 9258667ccc3e4ef8b26fe75531dd4591 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 20:46:19.994159 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:20.045747 4706 consensus_peers.cc:485] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:20.086918 4574 consensus_peers.cc:485] T 77dc51e6a85a4ddcb64db12b40c5810f P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:20.095424 6834 raft_consensus.cc:1261] T 747e7331d7ad4dd4b132672254037e1b P a4bf1df7fa6b4948be502c182bfbb833 [term 2 FOLLOWER]: Refusing update from remote peer c3d26eafd51547acb780d4aac5ebe7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 20:46:20.095818 12321 consensus_queue.cc:1035] T 747e7331d7ad4dd4b132672254037e1b P c3d26eafd51547acb780d4aac5ebe7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:20.097879 4377 raft_consensus.cc:3037] T ef4002126e8847809add182f262d62d9 P c92e6e46782849479c8a0c6d82276fb2 [term 1 LEARNER]: Advancing to term 2
W20240502 20:46:20.098740 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
I20240502 20:46:20.099182 4377 raft_consensus.cc:1261] T ef4002126e8847809add182f262d62d9 P c92e6e46782849479c8a0c6d82276fb2 [term 2 LEARNER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 20:46:20.099691 12286 consensus_queue.cc:1035] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 20:46:20.100003 4444 consensus_peers.cc:485] T 747e7331d7ad4dd4b132672254037e1b P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:20.101951 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:20.103228 12216 raft_consensus.cc:1050] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd: attempting to promote NON_VOTER c92e6e46782849479c8a0c6d82276fb2 to VOTER
W20240502 20:46:20.103303 12216 raft_consensus.cc:1053] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd: Unable to promote non-voter c92e6e46782849479c8a0c6d82276fb2: Illegal state: Leader has not yet committed an operation in its own term
I20240502 20:46:20.115597 12216 raft_consensus.cc:482] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:20.115734 12216 raft_consensus.cc:504] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:20.116160 12216 leader_election.cc:288] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
W20240502 20:46:20.117216 4706 leader_election.cc:334] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:20.117359 4703 leader_election.cc:334] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
I20240502 20:46:20.117450 4703 leader_election.cc:302] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed47c023452a497192f35d2f617bf9bd; no voters: 201d2b6485084e7dad2cbb722af94779, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:20.117619 12216 raft_consensus.cc:2726] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 20:46:20.122848 6768 consensus_peers.cc:485] T 71537846373844cc9ef396b79af60db4 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 20:46:20.148455 4507 raft_consensus.cc:1261] T 77dc51e6a85a4ddcb64db12b40c5810f P c3d26eafd51547acb780d4aac5ebe7cc [term 2 FOLLOWER]: Refusing update from remote peer 923cfab926e7443f8ffd7b21ad0a9d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 20:46:20.148828 12167 consensus_queue.cc:1035] T 77dc51e6a85a4ddcb64db12b40c5810f P 923cfab926e7443f8ffd7b21ad0a9d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 20:46:20.154768 6834 raft_consensus.cc:1261] T ef4002126e8847809add182f262d62d9 P a4bf1df7fa6b4948be502c182bfbb833 [term 2 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 20:46:20.155151 12286 consensus_queue.cc:1035] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:20.158358 12286 raft_consensus.cc:1050] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd: attempting to promote NON_VOTER c92e6e46782849479c8a0c6d82276fb2 to VOTER
I20240502 20:46:20.158782 12286 consensus_queue.cc:237] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: false } }
I20240502 20:46:20.159916 4377 raft_consensus.cc:1261] T ef4002126e8847809add182f262d62d9 P c92e6e46782849479c8a0c6d82276fb2 [term 2 LEARNER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 20:46:20.160241 6834 raft_consensus.cc:1261] T ef4002126e8847809add182f262d62d9 P a4bf1df7fa6b4948be502c182bfbb833 [term 2 FOLLOWER]: Refusing update from remote peer ed47c023452a497192f35d2f617bf9bd: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 20:46:20.160568 12216 consensus_queue.cc:1035] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 20:46:20.160702 4706 consensus_peers.cc:485] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:20.160732 12331 consensus_queue.cc:1035] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:20.162757 12216 raft_consensus.cc:2932] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: false } } }
I20240502 20:46:20.163332 6834 raft_consensus.cc:2932] T ef4002126e8847809add182f262d62d9 P a4bf1df7fa6b4948be502c182bfbb833 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: false } } }
I20240502 20:46:20.165138 4377 raft_consensus.cc:2932] T ef4002126e8847809add182f262d62d9 P c92e6e46782849479c8a0c6d82276fb2 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: false } } }
I20240502 20:46:20.165648 4230 catalog_manager.cc:5484] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd reported cstate change: config changed from index 5 to 7, c92e6e46782849479c8a0c6d82276fb2 (127.30.32.65) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "ed47c023452a497192f35d2f617bf9bd" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 20:46:20.303164 4441 consensus_peers.cc:485] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:20.347288 4441 consensus_peers.cc:485] T 6c8f977981c54fa1bf5d173e83608146 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:20.375229 4219 catalog_manager.cc:4573] TS 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): CreateTablet RPC failed for tablet 9ede0f8eef3440349ec5f0b93878c1f4: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:20.382536 4703 consensus_peers.cc:485] T 63f98288c8b1483493d5018c36e611bb P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:20.404484 4310 consensus_peers.cc:485] T a7ea68e8581a4cd5a6212abc96dfbcde P c92e6e46782849479c8a0c6d82276fb2 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:20.428194 6771 consensus_peers.cc:485] T a9afb75bfc114f59850eaea084a306f0 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:20.493476 4574 consensus_peers.cc:485] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:20.526369 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:20.562597 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:20.563853 4703 consensus_peers.cc:485] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:20.564293 4572 consensus_peers.cc:485] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:20.596536 6768 consensus_peers.cc:485] T 52e302882311488ab1cac90ad03fe270 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:20.606598 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:20.642804 4444 consensus_peers.cc:485] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:20.660208 4443 consensus_peers.cc:485] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:20.665462 6771 consensus_peers.cc:485] T 6201083694c344c8ad63720be89a3cdb P a4bf1df7fa6b4948be502c182bfbb833 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:20.671471 4312 consensus_peers.cc:485] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:20.697506 6768 consensus_peers.cc:485] T 6201083694c344c8ad63720be89a3cdb P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:20.719915 4705 consensus_peers.cc:485] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:20.725754 4310 consensus_peers.cc:485] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:20.734413 4312 consensus_peers.cc:485] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:20.757048 6770 consensus_peers.cc:485] T 71537846373844cc9ef396b79af60db4 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:20.786396 4572 consensus_peers.cc:485] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:20.802649 6770 consensus_peers.cc:485] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:20.809751 4443 consensus_peers.cc:485] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:20.813014 4703 consensus_peers.cc:485] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 20:46:20.833076 12167 raft_consensus.cc:482] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a47e5c56f13843018ac584914a8d4707)
I20240502 20:46:20.833204 12167 raft_consensus.cc:504] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true } }
I20240502 20:46:20.833616 12167 leader_election.cc:288] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
W20240502 20:46:20.834659 4574 leader_election.cc:334] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:20.834753 4575 leader_election.cc:334] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:20.834846 4575 leader_election.cc:302] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91; no voters: 7845c397230d49358ba2fad9db53f97c, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:20.834973 12167 raft_consensus.cc:2726] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:20.851491 4575 consensus_peers.cc:485] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:20.856969 6770 consensus_peers.cc:485] T 7ca68870fae3480786c84bd86dc4897e P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:20.897835 4443 consensus_peers.cc:485] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:20.901381 4574 consensus_peers.cc:485] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:20.905037 6771 consensus_peers.cc:485] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:20.906452 4574 consensus_peers.cc:485] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:20.906543 4575 consensus_peers.cc:485] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:20.921614 4312 consensus_peers.cc:485] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:20.928543 6768 consensus_peers.cc:485] T d37730b109d04b6395f75c7552524877 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:20.938509 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
I20240502 20:46:20.957258 12335 raft_consensus.cc:482] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 201d2b6485084e7dad2cbb722af94779)
I20240502 20:46:20.957437 12335 raft_consensus.cc:504] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } }
I20240502 20:46:20.957973 12335 leader_election.cc:288] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
W20240502 20:46:20.959004 4310 leader_election.cc:334] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:20.959101 4313 leader_election.cc:334] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:20.959187 4313 leader_election.cc:302] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2; no voters: 201d2b6485084e7dad2cbb722af94779, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:20.959332 12335 raft_consensus.cc:2726] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:20.981928 4706 consensus_peers.cc:485] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 20:46:20.984571 12167 raft_consensus.cc:482] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7845c397230d49358ba2fad9db53f97c)
W20240502 20:46:20.984637 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:20.984681 12167 raft_consensus.cc:504] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:20.985081 12167 leader_election.cc:288] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643), 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
W20240502 20:46:20.985960 4572 leader_election.cc:334] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:20.986974 4574 leader_election.cc:334] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
I20240502 20:46:20.987049 4574 leader_election.cc:302] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91; no voters: 201d2b6485084e7dad2cbb722af94779, 7845c397230d49358ba2fad9db53f97c
I20240502 20:46:20.987165 12167 raft_consensus.cc:2726] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:21.003912 6770 consensus_peers.cc:485] T dff789a7a50c4ff0a1eb29489ac7901a P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:21.007921 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:21.010830 4575 consensus_peers.cc:485] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 20:46:21.013104 12293 raft_consensus.cc:482] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7845c397230d49358ba2fad9db53f97c)
I20240502 20:46:21.013216 12293 raft_consensus.cc:504] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true } }
I20240502 20:46:21.013617 12293 leader_election.cc:288] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643), 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
W20240502 20:46:21.014753 6770 leader_election.cc:334] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:21.014837 6768 leader_election.cc:334] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
I20240502 20:46:21.014921 6768 leader_election.cc:302] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4bf1df7fa6b4948be502c182bfbb833; no voters: 201d2b6485084e7dad2cbb722af94779, 7845c397230d49358ba2fad9db53f97c
I20240502 20:46:21.015060 12293 raft_consensus.cc:2726] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:21.018815 4310 consensus_peers.cc:485] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:21.024969 4575 consensus_peers.cc:485] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 20:46:21.057348 12167 raft_consensus.cc:482] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7845c397230d49358ba2fad9db53f97c)
I20240502 20:46:21.057482 12167 raft_consensus.cc:504] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: true } }
W20240502 20:46:21.057683 4705 consensus_peers.cc:485] T 63f98288c8b1483493d5018c36e611bb P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 20:46:21.057937 12167 leader_election.cc:288] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
W20240502 20:46:21.058954 4572 leader_election.cc:334] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:21.059047 4574 leader_election.cc:334] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
I20240502 20:46:21.059118 4574 leader_election.cc:302] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91; no voters: 201d2b6485084e7dad2cbb722af94779, 7845c397230d49358ba2fad9db53f97c
I20240502 20:46:21.059267 12167 raft_consensus.cc:2726] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:21.060472 4441 consensus_peers.cc:485] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:21.070238 4443 consensus_peers.cc:485] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 20:46:21.083882 12335 raft_consensus.cc:482] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7845c397230d49358ba2fad9db53f97c)
I20240502 20:46:21.084017 12335 raft_consensus.cc:504] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: false } }
I20240502 20:46:21.084455 12335 leader_election.cc:288] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
W20240502 20:46:21.085538 4310 leader_election.cc:334] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:21.085634 4312 leader_election.cc:334] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
I20240502 20:46:21.085737 4312 leader_election.cc:302] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2; no voters: 201d2b6485084e7dad2cbb722af94779, 7845c397230d49358ba2fad9db53f97c
I20240502 20:46:21.085886 12335 raft_consensus.cc:2726] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 20:46:21.086648 12167 raft_consensus.cc:482] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7845c397230d49358ba2fad9db53f97c)
I20240502 20:46:21.086772 12167 raft_consensus.cc:504] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } }
I20240502 20:46:21.087129 12167 leader_election.cc:288] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
W20240502 20:46:21.088177 4572 leader_election.cc:334] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:21.088272 4574 leader_election.cc:334] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
I20240502 20:46:21.088352 4574 leader_election.cc:302] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91; no voters: 201d2b6485084e7dad2cbb722af94779, 7845c397230d49358ba2fad9db53f97c
I20240502 20:46:21.088500 12167 raft_consensus.cc:2726] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:21.096329 4444 consensus_peers.cc:485] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:21.157133 4313 consensus_peers.cc:485] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:21.166112 4444 consensus_peers.cc:485] T 28da7347f982468281570ffa7503a854 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:21.166503 4703 consensus_peers.cc:485] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:21.195338 6768 consensus_peers.cc:485] T c60371165de94a5d9ec1c7181f839e5b P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 20:46:21.219756 12335 raft_consensus.cc:482] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a47e5c56f13843018ac584914a8d4707)
I20240502 20:46:21.219906 12335 raft_consensus.cc:504] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:21.220451 12335 leader_election.cc:288] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
W20240502 20:46:21.221848 4313 leader_election.cc:334] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:21.221989 4312 leader_election.cc:334] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
I20240502 20:46:21.222082 4312 leader_election.cc:302] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2; no voters: 7845c397230d49358ba2fad9db53f97c, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:21.222214 12335 raft_consensus.cc:2726] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:21.285909 6768 consensus_peers.cc:485] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:21.312726 4575 consensus_peers.cc:485] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 20:46:21.350432 12335 raft_consensus.cc:482] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7845c397230d49358ba2fad9db53f97c)
I20240502 20:46:21.350574 12335 raft_consensus.cc:504] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:21.351006 12335 leader_election.cc:288] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
W20240502 20:46:21.352062 4312 leader_election.cc:334] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:21.352136 4313 leader_election.cc:334] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:21.352228 4313 leader_election.cc:302] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2; no voters: 7845c397230d49358ba2fad9db53f97c, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:21.352361 12335 raft_consensus.cc:2726] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 20:46:21.383735 12339 raft_consensus.cc:482] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a47e5c56f13843018ac584914a8d4707)
I20240502 20:46:21.383880 12339 raft_consensus.cc:504] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } }
I20240502 20:46:21.384323 12339 leader_election.cc:288] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
W20240502 20:46:21.385313 4705 leader_election.cc:334] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:21.385392 4706 leader_election.cc:334] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:21.385455 4706 leader_election.cc:302] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed47c023452a497192f35d2f617bf9bd; no voters: 7845c397230d49358ba2fad9db53f97c, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:21.385592 12339 raft_consensus.cc:2726] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:21.389073 6771 consensus_peers.cc:485] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 20:46:21.437368 12344 raft_consensus.cc:482] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a47e5c56f13843018ac584914a8d4707)
I20240502 20:46:21.437517 12344 raft_consensus.cc:504] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:21.437963 12344 leader_election.cc:288] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
W20240502 20:46:21.439282 4441 leader_election.cc:334] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:21.439396 4444 leader_election.cc:334] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:21.439477 4444 leader_election.cc:302] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3d26eafd51547acb780d4aac5ebe7cc; no voters: 201d2b6485084e7dad2cbb722af94779, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:21.439623 12344 raft_consensus.cc:2726] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 20:46:21.452517 12293 raft_consensus.cc:482] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 201d2b6485084e7dad2cbb722af94779)
I20240502 20:46:21.452677 12293 raft_consensus.cc:504] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } }
I20240502 20:46:21.453148 12293 leader_election.cc:288] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
W20240502 20:46:21.453953 6771 leader_election.cc:334] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:21.454079 6768 leader_election.cc:334] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
I20240502 20:46:21.454138 6768 leader_election.cc:302] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4bf1df7fa6b4948be502c182bfbb833; no voters: 201d2b6485084e7dad2cbb722af94779, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:21.454270 12293 raft_consensus.cc:2726] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 20:46:21.458053 12335 raft_consensus.cc:482] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a47e5c56f13843018ac584914a8d4707)
I20240502 20:46:21.458155 12335 raft_consensus.cc:504] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:21.458560 12335 leader_election.cc:288] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
W20240502 20:46:21.459419 4313 leader_election.cc:334] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:21.459513 4310 leader_election.cc:334] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
I20240502 20:46:21.459592 4310 leader_election.cc:302] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2; no voters: 201d2b6485084e7dad2cbb722af94779, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:21.459723 12335 raft_consensus.cc:2726] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:21.488444 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:21.515605 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:21.545717 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
I20240502 20:46:21.573573 12293 raft_consensus.cc:482] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a47e5c56f13843018ac584914a8d4707)
I20240502 20:46:21.573716 12293 raft_consensus.cc:504] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } }
I20240502 20:46:21.574203 12293 leader_election.cc:288] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
W20240502 20:46:21.575292 6770 leader_election.cc:334] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:21.575405 6771 leader_election.cc:334] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:21.575476 6771 leader_election.cc:302] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4bf1df7fa6b4948be502c182bfbb833; no voters: 7845c397230d49358ba2fad9db53f97c, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:21.575762 12293 raft_consensus.cc:2726] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:21.928189 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:21.947746 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:22.024698 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:22.136873 4703 consensus_peers.cc:485] T abe3bf6cb45a438fbd10bd5e78febc32 P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:22.180140 6768 consensus_peers.cc:485] T 6ed088a162a14d37b5da21ebb468a03b P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:22.314997 4443 consensus_peers.cc:485] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:22.334175 4572 consensus_peers.cc:485] T 51232fccd98c461fb3d9016081f52ba5 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:22.438930 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:22.442510 4219 catalog_manager.cc:4573] TS 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): CreateTablet RPC failed for tablet 9ede0f8eef3440349ec5f0b93878c1f4: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
I20240502 20:46:22.455560 12339 raft_consensus.cc:482] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:22.455694 12339 raft_consensus.cc:504] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:22.456144 12339 leader_election.cc:288] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
W20240502 20:46:22.457096 4703 leader_election.cc:334] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:22.457226 4706 leader_election.cc:334] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:22.457302 4706 leader_election.cc:302] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed47c023452a497192f35d2f617bf9bd; no voters: 201d2b6485084e7dad2cbb722af94779, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:22.457428 12339 raft_consensus.cc:2726] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 20:46:22.499300 4706 consensus_peers.cc:485] T 9258667ccc3e4ef8b26fe75531dd4591 P ed47c023452a497192f35d2f617bf9bd -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:22.518630 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:22.541946 6768 consensus_peers.cc:485] T 71537846373844cc9ef396b79af60db4 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:22.558053 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:22.613010 4444 consensus_peers.cc:485] T 747e7331d7ad4dd4b132672254037e1b P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:22.626735 4574 consensus_peers.cc:485] T 77dc51e6a85a4ddcb64db12b40c5810f P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:22.646126 4310 consensus_peers.cc:485] T a7ea68e8581a4cd5a6212abc96dfbcde P c92e6e46782849479c8a0c6d82276fb2 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:22.735666 4441 consensus_peers.cc:485] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:22.751456 4703 consensus_peers.cc:485] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:22.816255 6771 consensus_peers.cc:485] T a9afb75bfc114f59850eaea084a306f0 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:22.874446 4706 consensus_peers.cc:485] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:22.932024 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:22.942847 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:22.953719 4441 consensus_peers.cc:485] T 6c8f977981c54fa1bf5d173e83608146 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:22.955250 4574 consensus_peers.cc:485] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:22.961309 4703 consensus_peers.cc:485] T 63f98288c8b1483493d5018c36e611bb P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:23.040059 4705 consensus_peers.cc:485] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:23.079173 4310 consensus_peers.cc:485] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:23.080241 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
I20240502 20:46:23.091285 12293 raft_consensus.cc:482] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 201d2b6485084e7dad2cbb722af94779)
I20240502 20:46:23.091405 12293 raft_consensus.cc:504] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } }
I20240502 20:46:23.091779 12293 leader_election.cc:288] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
W20240502 20:46:23.092844 6768 leader_election.cc:334] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:23.093025 6771 leader_election.cc:334] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:23.093102 6771 leader_election.cc:302] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4bf1df7fa6b4948be502c182bfbb833; no voters: 201d2b6485084e7dad2cbb722af94779, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:23.093241 12293 raft_consensus.cc:2726] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:23.099329 4312 consensus_peers.cc:485] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:23.108495 4575 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111) [suppressed 143 similar messages]
W20240502 20:46:23.117313 6768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111) [suppressed 173 similar messages]
W20240502 20:46:23.118011 6768 consensus_peers.cc:485] T 6201083694c344c8ad63720be89a3cdb P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 20:46:23.124866 12293 raft_consensus.cc:482] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7845c397230d49358ba2fad9db53f97c)
I20240502 20:46:23.125007 12293 raft_consensus.cc:504] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true } }
I20240502 20:46:23.125443 12293 leader_election.cc:288] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643), 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
W20240502 20:46:23.125578 4313 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111) [suppressed 105 similar messages]
W20240502 20:46:23.126372 6768 leader_election.cc:334] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:23.126566 6770 leader_election.cc:334] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
I20240502 20:46:23.126637 6770 leader_election.cc:302] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4bf1df7fa6b4948be502c182bfbb833; no voters: 201d2b6485084e7dad2cbb722af94779, 7845c397230d49358ba2fad9db53f97c
I20240502 20:46:23.126782 12293 raft_consensus.cc:2726] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:23.150651 6771 consensus_peers.cc:485] T 6201083694c344c8ad63720be89a3cdb P a4bf1df7fa6b4948be502c182bfbb833 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:23.152011 4444 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111) [suppressed 122 similar messages]
W20240502 20:46:23.156448 4572 consensus_peers.cc:485] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:23.160008 4443 consensus_peers.cc:485] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:23.171710 4312 consensus_peers.cc:485] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:23.180032 6768 consensus_peers.cc:485] T 52e302882311488ab1cac90ad03fe270 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:23.192122 4312 consensus_peers.cc:485] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:23.202986 4703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111) [suppressed 124 similar messages]
W20240502 20:46:23.203619 4703 consensus_peers.cc:485] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:23.274008 6771 consensus_peers.cc:485] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:23.298444 6770 consensus_peers.cc:485] T 71537846373844cc9ef396b79af60db4 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:23.300102 4443 consensus_peers.cc:485] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:23.304634 4574 consensus_peers.cc:485] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:23.324242 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:23.335331 4222 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111) [suppressed 25 similar messages]
I20240502 20:46:23.346679 12348 raft_consensus.cc:482] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7845c397230d49358ba2fad9db53f97c)
I20240502 20:46:23.346793 12348 raft_consensus.cc:504] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:23.347184 12348 leader_election.cc:288] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
W20240502 20:46:23.348244 4313 leader_election.cc:334] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:23.348419 4312 leader_election.cc:334] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
I20240502 20:46:23.348479 4312 leader_election.cc:302] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2; no voters: 7845c397230d49358ba2fad9db53f97c, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:23.348611 12348 raft_consensus.cc:2726] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:23.352195 4572 consensus_peers.cc:485] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:23.364305 4574 consensus_peers.cc:485] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:23.370566 4444 consensus_peers.cc:485] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:23.440833 6770 consensus_peers.cc:485] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:23.465128 6770 consensus_peers.cc:485] T 7ca68870fae3480786c84bd86dc4897e P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:23.466692 6770 consensus_peers.cc:485] T dff789a7a50c4ff0a1eb29489ac7901a P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:23.471638 4575 consensus_peers.cc:485] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:23.495151 4706 consensus_peers.cc:485] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 20:46:23.495400 12346 raft_consensus.cc:482] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7845c397230d49358ba2fad9db53f97c)
I20240502 20:46:23.495507 12346 raft_consensus.cc:504] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:23.495882 12346 leader_election.cc:288] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643), 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
W20240502 20:46:23.496804 4574 leader_election.cc:334] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:23.496889 4572 leader_election.cc:334] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
I20240502 20:46:23.496950 4572 leader_election.cc:302] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91; no voters: 201d2b6485084e7dad2cbb722af94779, 7845c397230d49358ba2fad9db53f97c
I20240502 20:46:23.497113 12346 raft_consensus.cc:2726] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:23.502897 4310 consensus_peers.cc:485] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:23.513228 6768 consensus_peers.cc:485] T d37730b109d04b6395f75c7552524877 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:23.521402 4575 consensus_peers.cc:485] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:23.525836 4441 consensus_peers.cc:485] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:23.535907 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:23.546070 4575 consensus_peers.cc:485] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:23.548197 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:23.567337 6768 consensus_peers.cc:485] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 20:46:23.594267 12357 raft_consensus.cc:482] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7845c397230d49358ba2fad9db53f97c)
I20240502 20:46:23.594383 12357 raft_consensus.cc:504] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } }
I20240502 20:46:23.594707 12357 leader_election.cc:288] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
W20240502 20:46:23.595687 4574 leader_election.cc:334] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:23.595777 4572 leader_election.cc:334] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
I20240502 20:46:23.595862 4572 leader_election.cc:302] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91; no voters: 201d2b6485084e7dad2cbb722af94779, 7845c397230d49358ba2fad9db53f97c
I20240502 20:46:23.595988 12357 raft_consensus.cc:2726] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:23.617369 4575 consensus_peers.cc:485] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 20:46:23.634909 12348 raft_consensus.cc:482] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7845c397230d49358ba2fad9db53f97c)
I20240502 20:46:23.635022 12348 raft_consensus.cc:504] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: false } }
I20240502 20:46:23.635344 12348 leader_election.cc:288] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
W20240502 20:46:23.636253 4310 leader_election.cc:334] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:23.636345 4312 leader_election.cc:334] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
I20240502 20:46:23.636416 4312 leader_election.cc:302] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2; no voters: 201d2b6485084e7dad2cbb722af94779, 7845c397230d49358ba2fad9db53f97c
I20240502 20:46:23.636561 12348 raft_consensus.cc:2726] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:23.646276 4443 consensus_peers.cc:485] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:23.657739 4313 consensus_peers.cc:485] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:23.658929 4444 consensus_peers.cc:485] T 28da7347f982468281570ffa7503a854 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:23.665163 4444 consensus_peers.cc:485] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:23.665850 4705 consensus_peers.cc:485] T 63f98288c8b1483493d5018c36e611bb P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:23.683495 4443 consensus_peers.cc:485] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:23.718065 6768 consensus_peers.cc:485] T c60371165de94a5d9ec1c7181f839e5b P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:23.740345 4703 consensus_peers.cc:485] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:23.840325 4575 consensus_peers.cc:485] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:23.914314 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:23.979682 12363 raft_consensus.cc:482] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a47e5c56f13843018ac584914a8d4707)
I20240502 20:46:23.979818 12363 raft_consensus.cc:504] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:23.980206 12363 leader_election.cc:288] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
W20240502 20:46:23.981321 4444 leader_election.cc:334] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:23.981602 4441 leader_election.cc:334] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
I20240502 20:46:23.981679 4441 leader_election.cc:302] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3d26eafd51547acb780d4aac5ebe7cc; no voters: 201d2b6485084e7dad2cbb722af94779, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:23.981812 12363 raft_consensus.cc:2726] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 20:46:24.085150 12348 raft_consensus.cc:482] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a47e5c56f13843018ac584914a8d4707)
I20240502 20:46:24.085283 12348 raft_consensus.cc:504] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:24.085770 12348 leader_election.cc:288] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
W20240502 20:46:24.086954 4312 leader_election.cc:334] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:24.087038 4313 leader_election.cc:334] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:24.087087 4313 leader_election.cc:302] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2; no voters: 7845c397230d49358ba2fad9db53f97c, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:24.087349 12348 raft_consensus.cc:2726] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:24.108217 6771 consensus_peers.cc:485] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:24.151788 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
I20240502 20:46:24.163640 12346 raft_consensus.cc:482] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a47e5c56f13843018ac584914a8d4707)
I20240502 20:46:24.163780 12346 raft_consensus.cc:504] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true } }
I20240502 20:46:24.164252 12346 leader_election.cc:288] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
W20240502 20:46:24.164799 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:24.165271 4574 leader_election.cc:334] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:24.165364 4575 leader_election.cc:334] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:24.165445 4575 leader_election.cc:302] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91; no voters: 7845c397230d49358ba2fad9db53f97c, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:24.165581 12346 raft_consensus.cc:2726] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 20:46:24.235596 12348 raft_consensus.cc:482] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 201d2b6485084e7dad2cbb722af94779)
I20240502 20:46:24.235746 12348 raft_consensus.cc:504] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } }
I20240502 20:46:24.236140 12348 leader_election.cc:288] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
W20240502 20:46:24.237143 4313 leader_election.cc:334] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:24.237222 4310 leader_election.cc:334] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
I20240502 20:46:24.237303 4313 leader_election.cc:302] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2; no voters: 201d2b6485084e7dad2cbb722af94779, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:24.237449 12348 raft_consensus.cc:2726] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 20:46:24.308025 12346 raft_consensus.cc:482] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7845c397230d49358ba2fad9db53f97c)
I20240502 20:46:24.308148 12346 raft_consensus.cc:504] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: true } }
I20240502 20:46:24.308550 12346 leader_election.cc:288] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
W20240502 20:46:24.309441 4574 leader_election.cc:334] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:24.309536 4572 leader_election.cc:334] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
I20240502 20:46:24.309621 4572 leader_election.cc:302] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91; no voters: 201d2b6485084e7dad2cbb722af94779, 7845c397230d49358ba2fad9db53f97c
I20240502 20:46:24.309754 12346 raft_consensus.cc:2726] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 20:46:24.480451 12367 raft_consensus.cc:482] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a47e5c56f13843018ac584914a8d4707)
I20240502 20:46:24.480579 12367 raft_consensus.cc:504] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } }
I20240502 20:46:24.480994 12367 leader_election.cc:288] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
W20240502 20:46:24.481894 4705 leader_election.cc:334] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:24.481981 4706 leader_election.cc:334] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:24.482074 4706 leader_election.cc:302] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed47c023452a497192f35d2f617bf9bd; no voters: 7845c397230d49358ba2fad9db53f97c, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:24.482244 12367 raft_consensus.cc:2726] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 20:46:24.496508 12293 raft_consensus.cc:482] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a47e5c56f13843018ac584914a8d4707)
I20240502 20:46:24.496613 12293 raft_consensus.cc:504] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } }
I20240502 20:46:24.496935 12293 leader_election.cc:288] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
W20240502 20:46:24.497817 6771 leader_election.cc:334] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:24.497949 6770 leader_election.cc:334] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
I20240502 20:46:24.498018 6770 leader_election.cc:302] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4bf1df7fa6b4948be502c182bfbb833; no voters: 7845c397230d49358ba2fad9db53f97c, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:24.498145 12293 raft_consensus.cc:2726] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:24.529757 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:24.638761 4703 consensus_peers.cc:485] T abe3bf6cb45a438fbd10bd5e78febc32 P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 20:46:24.646544 12348 raft_consensus.cc:482] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a47e5c56f13843018ac584914a8d4707)
I20240502 20:46:24.646654 12348 raft_consensus.cc:504] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:24.647055 12348 leader_election.cc:288] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
W20240502 20:46:24.648036 4313 leader_election.cc:334] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:24.648123 4310 leader_election.cc:334] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
I20240502 20:46:24.648188 4310 leader_election.cc:302] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2; no voters: 201d2b6485084e7dad2cbb722af94779, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:24.648339 12348 raft_consensus.cc:2726] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:24.679080 6768 consensus_peers.cc:485] T 6ed088a162a14d37b5da21ebb468a03b P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:24.697723 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:24.711956 4443 consensus_peers.cc:485] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:24.766947 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:24.798151 4572 consensus_peers.cc:485] T 51232fccd98c461fb3d9016081f52ba5 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:24.899672 4441 consensus_peers.cc:485] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:24.907966 4310 consensus_peers.cc:485] T a7ea68e8581a4cd5a6212abc96dfbcde P c92e6e46782849479c8a0c6d82276fb2 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:24.913867 4706 consensus_peers.cc:485] T 9258667ccc3e4ef8b26fe75531dd4591 P ed47c023452a497192f35d2f617bf9bd -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:24.963024 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:25.030304 4444 consensus_peers.cc:485] T 747e7331d7ad4dd4b132672254037e1b P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:25.124614 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:25.172078 4703 consensus_peers.cc:485] T 63f98288c8b1483493d5018c36e611bb P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:25.176926 4574 consensus_peers.cc:485] T 77dc51e6a85a4ddcb64db12b40c5810f P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:25.186290 6768 consensus_peers.cc:485] T 71537846373844cc9ef396b79af60db4 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:25.186489 4441 consensus_peers.cc:485] T 6c8f977981c54fa1bf5d173e83608146 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 20:46:25.194176 12357 raft_consensus.cc:482] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7845c397230d49358ba2fad9db53f97c)
I20240502 20:46:25.194334 12357 raft_consensus.cc:504] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } }
I20240502 20:46:25.194721 12357 leader_election.cc:288] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
W20240502 20:46:25.195657 4574 leader_election.cc:334] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:25.195801 4572 leader_election.cc:334] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
I20240502 20:46:25.195876 4572 leader_election.cc:302] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91; no voters: 201d2b6485084e7dad2cbb722af94779, 7845c397230d49358ba2fad9db53f97c
I20240502 20:46:25.196003 12357 raft_consensus.cc:2726] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:25.258682 4703 consensus_peers.cc:485] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:25.268759 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:25.300451 6768 consensus_peers.cc:485] T 6201083694c344c8ad63720be89a3cdb P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:25.387780 4574 consensus_peers.cc:485] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:25.407390 6771 consensus_peers.cc:485] T a9afb75bfc114f59850eaea084a306f0 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:25.413518 6771 consensus_peers.cc:485] T 6201083694c344c8ad63720be89a3cdb P a4bf1df7fa6b4948be502c182bfbb833 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:25.422116 4310 consensus_peers.cc:485] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:25.446552 4703 consensus_peers.cc:485] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:25.453253 4706 consensus_peers.cc:485] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:25.465216 4705 consensus_peers.cc:485] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:25.500151 4443 consensus_peers.cc:485] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:25.531507 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:25.584614 4312 consensus_peers.cc:485] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:25.589993 4312 consensus_peers.cc:485] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:25.644280 4312 consensus_peers.cc:485] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:25.651008 4572 consensus_peers.cc:485] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:25.671378 4444 consensus_peers.cc:485] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:25.680125 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:25.680852 4575 consensus_peers.cc:485] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:25.692028 6770 consensus_peers.cc:485] T 71537846373844cc9ef396b79af60db4 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:25.707707 6768 consensus_peers.cc:485] T 52e302882311488ab1cac90ad03fe270 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:25.721221 4705 consensus_peers.cc:485] T 63f98288c8b1483493d5018c36e611bb P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:25.742116 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:25.779654 4310 consensus_peers.cc:485] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:25.811041 4443 consensus_peers.cc:485] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:25.817399 4572 consensus_peers.cc:485] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 20:46:25.847357 12293 raft_consensus.cc:482] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 201d2b6485084e7dad2cbb722af94779)
I20240502 20:46:25.847476 12293 raft_consensus.cc:504] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } }
I20240502 20:46:25.847831 12293 leader_election.cc:288] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
W20240502 20:46:25.849023 6771 leader_election.cc:334] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:25.849161 6768 leader_election.cc:334] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
I20240502 20:46:25.849217 6768 leader_election.cc:302] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4bf1df7fa6b4948be502c182bfbb833; no voters: 201d2b6485084e7dad2cbb722af94779, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:25.849498 12293 raft_consensus.cc:2726] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:25.854688 6768 consensus_peers.cc:485] T d37730b109d04b6395f75c7552524877 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:25.935322 4574 consensus_peers.cc:485] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:25.964524 4574 consensus_peers.cc:485] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:25.965566 4703 consensus_peers.cc:485] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:25.980763 4575 consensus_peers.cc:485] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:25.994602 4443 consensus_peers.cc:485] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:26.012233 6770 consensus_peers.cc:485] T dff789a7a50c4ff0a1eb29489ac7901a P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:26.014082 6770 consensus_peers.cc:485] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:26.020246 4575 consensus_peers.cc:485] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:26.038947 4444 consensus_peers.cc:485] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:26.059897 4706 consensus_peers.cc:485] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:26.067262 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:26.078294 4313 consensus_peers.cc:485] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:26.089706 4575 consensus_peers.cc:485] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:26.102780 4441 consensus_peers.cc:485] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:26.118398 6771 consensus_peers.cc:485] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:26.133527 6768 consensus_peers.cc:485] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:26.178251 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:26.184939 6770 consensus_peers.cc:485] T 7ca68870fae3480786c84bd86dc4897e P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:26.210582 4443 consensus_peers.cc:485] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:26.267758 4575 consensus_peers.cc:485] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 20:46:26.288106 12378 raft_consensus.cc:482] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:26.288242 12378 raft_consensus.cc:504] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:26.288729 12378 leader_election.cc:288] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
W20240502 20:46:26.289898 4703 leader_election.cc:334] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:26.290014 4706 leader_election.cc:334] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:26.290096 4706 leader_election.cc:302] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed47c023452a497192f35d2f617bf9bd; no voters: 201d2b6485084e7dad2cbb722af94779, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:26.290269 12378 raft_consensus.cc:2726] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 20:46:26.294976 12293 raft_consensus.cc:482] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7845c397230d49358ba2fad9db53f97c)
I20240502 20:46:26.295109 12293 raft_consensus.cc:504] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true } }
I20240502 20:46:26.295639 12293 leader_election.cc:288] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643), 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
W20240502 20:46:26.295653 4444 consensus_peers.cc:485] T 28da7347f982468281570ffa7503a854 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:26.296911 6770 leader_election.cc:334] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:26.297024 6768 leader_election.cc:334] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
I20240502 20:46:26.297108 6768 leader_election.cc:302] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4bf1df7fa6b4948be502c182bfbb833; no voters: 201d2b6485084e7dad2cbb722af94779, 7845c397230d49358ba2fad9db53f97c
I20240502 20:46:26.297245 12293 raft_consensus.cc:2726] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:26.312446 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:26.353266 6768 consensus_peers.cc:485] T c60371165de94a5d9ec1c7181f839e5b P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 20:46:26.361116 12376 raft_consensus.cc:482] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7845c397230d49358ba2fad9db53f97c)
I20240502 20:46:26.361246 12376 raft_consensus.cc:504] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:26.361665 12376 leader_election.cc:288] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
W20240502 20:46:26.362625 4313 leader_election.cc:334] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:26.362746 4312 leader_election.cc:334] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
I20240502 20:46:26.362830 4312 leader_election.cc:302] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2; no voters: 7845c397230d49358ba2fad9db53f97c, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:26.362960 12376 raft_consensus.cc:2726] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:26.403509 6771 consensus_peers.cc:485] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 20:46:26.558480 12376 raft_consensus.cc:482] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a47e5c56f13843018ac584914a8d4707)
I20240502 20:46:26.558607 12376 raft_consensus.cc:504] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:26.558960 12376 leader_election.cc:288] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
W20240502 20:46:26.559891 4312 leader_election.cc:334] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:26.560045 4313 leader_election.cc:334] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:26.560115 4313 leader_election.cc:302] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2; no voters: 7845c397230d49358ba2fad9db53f97c, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:26.560251 12376 raft_consensus.cc:2726] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:26.561659 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:26.569528 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:26.570207 4219 catalog_manager.cc:4573] TS 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): CreateTablet RPC failed for tablet 9ede0f8eef3440349ec5f0b93878c1f4: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:26.755263 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
I20240502 20:46:26.764631 12373 raft_consensus.cc:482] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7845c397230d49358ba2fad9db53f97c)
I20240502 20:46:26.764750 12373 raft_consensus.cc:504] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:26.765163 12373 leader_election.cc:288] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643), 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
W20240502 20:46:26.766124 4572 leader_election.cc:334] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:26.766283 4574 leader_election.cc:334] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
I20240502 20:46:26.766353 4574 leader_election.cc:302] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91; no voters: 201d2b6485084e7dad2cbb722af94779, 7845c397230d49358ba2fad9db53f97c
I20240502 20:46:26.766475 12373 raft_consensus.cc:2726] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 20:46:26.960621 12376 raft_consensus.cc:482] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 201d2b6485084e7dad2cbb722af94779)
I20240502 20:46:26.960752 12376 raft_consensus.cc:504] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } }
I20240502 20:46:26.961131 12376 leader_election.cc:288] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
W20240502 20:46:26.962003 4310 leader_election.cc:334] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:26.962085 4313 leader_election.cc:334] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:26.962163 4313 leader_election.cc:302] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2; no voters: 201d2b6485084e7dad2cbb722af94779, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:26.962322 12376 raft_consensus.cc:2726] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:26.982286 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:27.024176 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
I20240502 20:46:27.051447 12387 raft_consensus.cc:482] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a47e5c56f13843018ac584914a8d4707)
I20240502 20:46:27.051559 12387 raft_consensus.cc:504] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:27.051934 12387 leader_election.cc:288] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
W20240502 20:46:27.052804 4441 leader_election.cc:334] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:27.053102 4444 leader_election.cc:334] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:27.053156 4444 leader_election.cc:302] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3d26eafd51547acb780d4aac5ebe7cc; no voters: 201d2b6485084e7dad2cbb722af94779, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:27.053288 12387 raft_consensus.cc:2726] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:27.062631 6768 consensus_peers.cc:485] T 6ed088a162a14d37b5da21ebb468a03b P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:27.142117 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:27.162974 4703 consensus_peers.cc:485] T abe3bf6cb45a438fbd10bd5e78febc32 P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:27.220368 4572 consensus_peers.cc:485] T 51232fccd98c461fb3d9016081f52ba5 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:27.221292 4706 consensus_peers.cc:485] T 9258667ccc3e4ef8b26fe75531dd4591 P ed47c023452a497192f35d2f617bf9bd -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:27.229426 4443 consensus_peers.cc:485] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:27.335947 4441 consensus_peers.cc:485] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:27.421940 4703 consensus_peers.cc:485] T 63f98288c8b1483493d5018c36e611bb P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 20:46:27.471107 12373 raft_consensus.cc:482] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a47e5c56f13843018ac584914a8d4707)
I20240502 20:46:27.471253 12373 raft_consensus.cc:504] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true } }
I20240502 20:46:27.471704 12373 leader_election.cc:288] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
W20240502 20:46:27.472862 4575 leader_election.cc:334] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:27.472962 4574 leader_election.cc:334] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
I20240502 20:46:27.473074 4574 leader_election.cc:302] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91; no voters: 7845c397230d49358ba2fad9db53f97c, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:27.473235 12373 raft_consensus.cc:2726] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:27.485623 4574 consensus_peers.cc:485] T 77dc51e6a85a4ddcb64db12b40c5810f P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 20:46:27.489377 12373 raft_consensus.cc:482] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7845c397230d49358ba2fad9db53f97c)
I20240502 20:46:27.489481 12373 raft_consensus.cc:504] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: true } }
I20240502 20:46:27.489799 12373 leader_election.cc:288] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
W20240502 20:46:27.490739 4574 leader_election.cc:334] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:27.490854 4572 leader_election.cc:334] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
I20240502 20:46:27.490917 4572 leader_election.cc:302] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91; no voters: 201d2b6485084e7dad2cbb722af94779, 7845c397230d49358ba2fad9db53f97c
I20240502 20:46:27.491047 12373 raft_consensus.cc:2726] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:27.537273 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:27.552237 4703 consensus_peers.cc:485] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:27.553583 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:27.582365 12382 raft_consensus.cc:482] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a47e5c56f13843018ac584914a8d4707)
I20240502 20:46:27.582504 12382 raft_consensus.cc:504] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } }
I20240502 20:46:27.582890 12382 leader_election.cc:288] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
W20240502 20:46:27.583801 6771 leader_election.cc:334] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:27.583874 6770 leader_election.cc:334] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
I20240502 20:46:27.583967 6770 leader_election.cc:302] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4bf1df7fa6b4948be502c182bfbb833; no voters: 7845c397230d49358ba2fad9db53f97c, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:27.584113 12382 raft_consensus.cc:2726] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:27.589010 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:27.649962 4444 consensus_peers.cc:485] T 747e7331d7ad4dd4b132672254037e1b P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:27.677973 4310 consensus_peers.cc:485] T a7ea68e8581a4cd5a6212abc96dfbcde P c92e6e46782849479c8a0c6d82276fb2 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:27.718818 6768 consensus_peers.cc:485] T 71537846373844cc9ef396b79af60db4 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 20:46:27.729943 12393 raft_consensus.cc:482] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a47e5c56f13843018ac584914a8d4707)
I20240502 20:46:27.730110 12393 raft_consensus.cc:504] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } }
I20240502 20:46:27.730541 12393 leader_election.cc:288] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
W20240502 20:46:27.731580 4705 leader_election.cc:334] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:27.731674 4706 leader_election.cc:334] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:27.731766 4706 leader_election.cc:302] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed47c023452a497192f35d2f617bf9bd; no voters: 7845c397230d49358ba2fad9db53f97c, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:27.731922 12393 raft_consensus.cc:2726] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:27.736356 4441 consensus_peers.cc:485] T 6c8f977981c54fa1bf5d173e83608146 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:27.753612 6768 consensus_peers.cc:485] T 6201083694c344c8ad63720be89a3cdb P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:27.818167 4443 consensus_peers.cc:485] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:27.827286 4574 consensus_peers.cc:485] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:27.937665 4703 consensus_peers.cc:485] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:27.940256 4444 consensus_peers.cc:485] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:27.944814 6771 consensus_peers.cc:485] T 6201083694c344c8ad63720be89a3cdb P a4bf1df7fa6b4948be502c182bfbb833 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:27.960835 4706 consensus_peers.cc:485] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:27.970873 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:27.979678 4312 consensus_peers.cc:485] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 20:46:27.984225 12382 raft_consensus.cc:482] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 201d2b6485084e7dad2cbb722af94779)
I20240502 20:46:27.984364 12382 raft_consensus.cc:504] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } }
I20240502 20:46:27.984797 12382 leader_election.cc:288] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
W20240502 20:46:27.985805 6768 leader_election.cc:334] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:27.985888 6771 leader_election.cc:334] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:27.985958 6771 leader_election.cc:302] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4bf1df7fa6b4948be502c182bfbb833; no voters: 201d2b6485084e7dad2cbb722af94779, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:27.986096 12382 raft_consensus.cc:2726] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:28.011425 6770 consensus_peers.cc:485] T 71537846373844cc9ef396b79af60db4 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:28.029172 4310 consensus_peers.cc:485] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:28.046061 4705 consensus_peers.cc:485] T 63f98288c8b1483493d5018c36e611bb P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 20:46:28.082674 12396 raft_consensus.cc:482] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7845c397230d49358ba2fad9db53f97c)
I20240502 20:46:28.082851 12396 raft_consensus.cc:504] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: false } }
I20240502 20:46:28.083315 12396 leader_election.cc:288] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
W20240502 20:46:28.084355 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:28.084353 4312 leader_election.cc:334] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:28.084440 4310 leader_election.cc:334] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
I20240502 20:46:28.084527 4310 leader_election.cc:302] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2; no voters: 201d2b6485084e7dad2cbb722af94779, 7845c397230d49358ba2fad9db53f97c
I20240502 20:46:28.084656 12396 raft_consensus.cc:2726] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:28.098039 4575 consensus_peers.cc:485] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:28.119804 6768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111) [suppressed 163 similar messages]
W20240502 20:46:28.122771 6771 consensus_peers.cc:485] T a9afb75bfc114f59850eaea084a306f0 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:28.132143 4312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111) [suppressed 97 similar messages]
W20240502 20:46:28.132788 4312 consensus_peers.cc:485] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:28.158854 4444 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111) [suppressed 126 similar messages]
W20240502 20:46:28.187919 6768 consensus_peers.cc:485] T 52e302882311488ab1cac90ad03fe270 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:28.195566 4705 consensus_peers.cc:485] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:28.206899 6768 consensus_peers.cc:485] T d37730b109d04b6395f75c7552524877 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:28.210008 4705 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240502 20:46:28.210768 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:28.212888 4443 consensus_peers.cc:485] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:28.260442 4312 consensus_peers.cc:485] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:28.290599 4572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111) [suppressed 149 similar messages]
W20240502 20:46:28.291265 4572 consensus_peers.cc:485] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:28.371505 4572 consensus_peers.cc:485] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:28.380789 6770 consensus_peers.cc:485] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:28.388779 4444 consensus_peers.cc:485] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:28.397262 4575 consensus_peers.cc:485] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:28.408543 4310 consensus_peers.cc:485] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 20:46:28.410928 12396 raft_consensus.cc:482] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a47e5c56f13843018ac584914a8d4707)
I20240502 20:46:28.411052 12396 raft_consensus.cc:504] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:28.411473 12396 leader_election.cc:288] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
W20240502 20:46:28.412461 4310 leader_election.cc:334] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:28.412547 4313 leader_election.cc:334] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:28.412664 4313 leader_election.cc:302] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2; no voters: 201d2b6485084e7dad2cbb722af94779, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:28.412811 12396 raft_consensus.cc:2726] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:28.414469 4443 consensus_peers.cc:485] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:28.456204 4574 consensus_peers.cc:485] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:28.487920 4574 consensus_peers.cc:485] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:28.500504 4575 consensus_peers.cc:485] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:28.506429 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:28.521955 6770 consensus_peers.cc:485] T dff789a7a50c4ff0a1eb29489ac7901a P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:28.546563 4575 consensus_peers.cc:485] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:28.550328 6771 consensus_peers.cc:485] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:28.553283 4441 consensus_peers.cc:485] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:28.560592 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:28.597270 6771 consensus_peers.cc:485] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:28.647766 4444 consensus_peers.cc:485] T 28da7347f982468281570ffa7503a854 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:28.669670 4703 consensus_peers.cc:485] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:28.672689 4313 consensus_peers.cc:485] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:28.692626 4706 consensus_peers.cc:485] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:28.696939 6768 consensus_peers.cc:485] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:28.731753 4575 consensus_peers.cc:485] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:28.824862 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:28.855933 6768 consensus_peers.cc:485] T c60371165de94a5d9ec1c7181f839e5b P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:28.906076 4443 consensus_peers.cc:485] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:28.916555 6770 consensus_peers.cc:485] T 7ca68870fae3480786c84bd86dc4897e P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:28.925496 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:28.937345 12382 raft_consensus.cc:482] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7845c397230d49358ba2fad9db53f97c)
I20240502 20:46:28.937462 12382 raft_consensus.cc:504] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true } }
I20240502 20:46:28.937788 12382 leader_election.cc:288] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643), 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
W20240502 20:46:28.938638 6768 leader_election.cc:334] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:28.938724 6770 leader_election.cc:334] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
I20240502 20:46:28.938793 6770 leader_election.cc:302] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4bf1df7fa6b4948be502c182bfbb833; no voters: 201d2b6485084e7dad2cbb722af94779, 7845c397230d49358ba2fad9db53f97c
I20240502 20:46:28.938920 12382 raft_consensus.cc:2726] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:29.088923 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
I20240502 20:46:29.147202 12396 raft_consensus.cc:482] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 201d2b6485084e7dad2cbb722af94779)
I20240502 20:46:29.147349 12396 raft_consensus.cc:504] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } }
I20240502 20:46:29.147776 12396 leader_election.cc:288] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
W20240502 20:46:29.148708 4310 leader_election.cc:334] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:29.148844 4313 leader_election.cc:334] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:29.148924 4313 leader_election.cc:302] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2; no voters: 201d2b6485084e7dad2cbb722af94779, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:29.149075 12396 raft_consensus.cc:2726] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:29.286773 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:29.405553 4703 consensus_peers.cc:485] T abe3bf6cb45a438fbd10bd5e78febc32 P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:29.516397 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:29.608261 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:29.609405 6768 consensus_peers.cc:485] T 6ed088a162a14d37b5da21ebb468a03b P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:29.693120 4443 consensus_peers.cc:485] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:29.713089 4572 consensus_peers.cc:485] T 51232fccd98c461fb3d9016081f52ba5 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:29.827178 4703 consensus_peers.cc:485] T 63f98288c8b1483493d5018c36e611bb P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:29.876348 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:29.907274 4706 consensus_peers.cc:485] T 9258667ccc3e4ef8b26fe75531dd4591 P ed47c023452a497192f35d2f617bf9bd -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:29.909922 4310 consensus_peers.cc:485] T a7ea68e8581a4cd5a6212abc96dfbcde P c92e6e46782849479c8a0c6d82276fb2 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:29.951617 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:30.103492 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:30.121824 4441 consensus_peers.cc:485] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:30.137142 4444 consensus_peers.cc:485] T 747e7331d7ad4dd4b132672254037e1b P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:30.142144 6768 consensus_peers.cc:485] T 6201083694c344c8ad63720be89a3cdb P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:30.143754 4574 consensus_peers.cc:485] T 77dc51e6a85a4ddcb64db12b40c5810f P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:30.236061 4706 consensus_peers.cc:485] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:30.239403 4703 consensus_peers.cc:485] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:30.244781 4312 consensus_peers.cc:485] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:30.325618 4312 consensus_peers.cc:485] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 20:46:30.340021 12396 raft_consensus.cc:482] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a47e5c56f13843018ac584914a8d4707)
I20240502 20:46:30.340152 12396 raft_consensus.cc:504] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:30.340561 12396 leader_election.cc:288] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
W20240502 20:46:30.341480 4312 leader_election.cc:334] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:30.341626 4313 leader_election.cc:334] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:30.341710 4313 leader_election.cc:302] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2; no voters: 7845c397230d49358ba2fad9db53f97c, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:30.341885 12396 raft_consensus.cc:2726] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:30.344067 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:30.344669 4444 consensus_peers.cc:485] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:30.350795 4441 consensus_peers.cc:485] T 6c8f977981c54fa1bf5d173e83608146 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:30.351469 6768 consensus_peers.cc:485] T 71537846373844cc9ef396b79af60db4 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:30.446658 4703 consensus_peers.cc:485] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:30.472452 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:30.511767 6771 consensus_peers.cc:485] T 6201083694c344c8ad63720be89a3cdb P a4bf1df7fa6b4948be502c182bfbb833 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:30.536567 4443 consensus_peers.cc:485] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:30.565101 4574 consensus_peers.cc:485] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:30.574612 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:30.601008 4705 consensus_peers.cc:485] T 63f98288c8b1483493d5018c36e611bb P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:30.687410 4575 consensus_peers.cc:485] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:30.698221 6770 consensus_peers.cc:485] T 71537846373844cc9ef396b79af60db4 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:30.717283 4575 consensus_peers.cc:485] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:30.727294 6768 consensus_peers.cc:485] T 52e302882311488ab1cac90ad03fe270 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:30.741150 4443 consensus_peers.cc:485] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:30.774287 4443 consensus_peers.cc:485] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:30.790807 4705 consensus_peers.cc:485] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:30.803305 6771 consensus_peers.cc:485] T a9afb75bfc114f59850eaea084a306f0 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:30.815819 4312 consensus_peers.cc:485] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:30.828809 6770 consensus_peers.cc:485] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:30.839337 6768 consensus_peers.cc:485] T d37730b109d04b6395f75c7552524877 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:30.844009 4575 consensus_peers.cc:485] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:30.883337 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:30.885784 4310 consensus_peers.cc:485] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:30.892000 4572 consensus_peers.cc:485] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:30.900146 4574 consensus_peers.cc:485] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:30.947001 4441 consensus_peers.cc:485] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:30.959944 6771 consensus_peers.cc:485] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:30.962718 4706 consensus_peers.cc:485] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:30.974288 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:30.993778 4310 consensus_peers.cc:485] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:30.995422 4443 consensus_peers.cc:485] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:31.008179 4313 consensus_peers.cc:485] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 20:46:31.018071 12406 raft_consensus.cc:482] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7845c397230d49358ba2fad9db53f97c)
I20240502 20:46:31.018209 12406 raft_consensus.cc:504] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } }
I20240502 20:46:31.018618 12406 leader_election.cc:288] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
W20240502 20:46:31.019959 4574 leader_election.cc:334] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:31.020068 4572 leader_election.cc:334] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
I20240502 20:46:31.020128 4572 leader_election.cc:302] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91; no voters: 201d2b6485084e7dad2cbb722af94779, 7845c397230d49358ba2fad9db53f97c
I20240502 20:46:31.020277 12406 raft_consensus.cc:2726] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:31.041918 6771 consensus_peers.cc:485] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:31.082370 4572 consensus_peers.cc:485] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:31.138334 6770 consensus_peers.cc:485] T dff789a7a50c4ff0a1eb29489ac7901a P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:31.149160 4574 consensus_peers.cc:485] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:31.152817 6768 consensus_peers.cc:485] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:31.163093 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
I20240502 20:46:31.184906 12414 raft_consensus.cc:482] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a47e5c56f13843018ac584914a8d4707)
I20240502 20:46:31.185106 12414 raft_consensus.cc:504] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } }
I20240502 20:46:31.185568 12414 leader_election.cc:288] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
W20240502 20:46:31.186651 6770 leader_election.cc:334] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:31.186739 6771 leader_election.cc:334] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:31.186821 6771 leader_election.cc:302] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4bf1df7fa6b4948be502c182bfbb833; no voters: 7845c397230d49358ba2fad9db53f97c, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:31.186959 12414 raft_consensus.cc:2726] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:31.206702 4444 consensus_peers.cc:485] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:31.216856 4444 consensus_peers.cc:485] T 28da7347f982468281570ffa7503a854 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:31.260121 4703 consensus_peers.cc:485] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:31.309916 4575 consensus_peers.cc:485] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:31.321096 6770 consensus_peers.cc:485] T 7ca68870fae3480786c84bd86dc4897e P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 20:46:31.427819 12404 raft_consensus.cc:482] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a47e5c56f13843018ac584914a8d4707)
I20240502 20:46:31.427981 12404 raft_consensus.cc:504] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:31.428455 12404 leader_election.cc:288] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
W20240502 20:46:31.429570 4441 leader_election.cc:334] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:31.429733 4444 leader_election.cc:334] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:31.429798 4444 leader_election.cc:302] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3d26eafd51547acb780d4aac5ebe7cc; no voters: 201d2b6485084e7dad2cbb722af94779, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:31.429922 12404 raft_consensus.cc:2726] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:31.438532 4575 consensus_peers.cc:485] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:31.446400 6768 consensus_peers.cc:485] T c60371165de94a5d9ec1c7181f839e5b P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:31.507550 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:31.543814 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:31.740123 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:31.800087 4703 consensus_peers.cc:485] T abe3bf6cb45a438fbd10bd5e78febc32 P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:31.963641 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:32.143088 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:32.209333 4572 consensus_peers.cc:485] T 51232fccd98c461fb3d9016081f52ba5 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:32.213588 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:32.241567 4443 consensus_peers.cc:485] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:32.272414 4310 consensus_peers.cc:485] T a7ea68e8581a4cd5a6212abc96dfbcde P c92e6e46782849479c8a0c6d82276fb2 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:32.311254 6768 consensus_peers.cc:485] T 6ed088a162a14d37b5da21ebb468a03b P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:32.338726 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:32.518872 4703 consensus_peers.cc:485] T 63f98288c8b1483493d5018c36e611bb P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:32.560438 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:32.616300 4574 consensus_peers.cc:485] T 77dc51e6a85a4ddcb64db12b40c5810f P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:32.646212 4703 consensus_peers.cc:485] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:32.662679 4444 consensus_peers.cc:485] T 747e7331d7ad4dd4b132672254037e1b P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:32.678072 4312 consensus_peers.cc:485] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:32.710415 4312 consensus_peers.cc:485] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:32.723986 4441 consensus_peers.cc:485] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 20:46:32.724704 12396 raft_consensus.cc:482] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7845c397230d49358ba2fad9db53f97c)
I20240502 20:46:32.724795 12396 raft_consensus.cc:504] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:32.725173 12396 leader_election.cc:288] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
W20240502 20:46:32.726042 4313 leader_election.cc:334] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:32.726148 4312 leader_election.cc:334] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
I20240502 20:46:32.726233 4312 leader_election.cc:302] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2; no voters: 7845c397230d49358ba2fad9db53f97c, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:32.726356 12396 raft_consensus.cc:2726] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:32.748034 4703 consensus_peers.cc:485] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:32.769642 4706 consensus_peers.cc:485] T 9258667ccc3e4ef8b26fe75531dd4591 P ed47c023452a497192f35d2f617bf9bd -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:32.808097 6768 consensus_peers.cc:485] T 71537846373844cc9ef396b79af60db4 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:32.811379 4444 consensus_peers.cc:485] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:32.838109 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:32.839303 6768 consensus_peers.cc:485] T 6201083694c344c8ad63720be89a3cdb P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:32.885149 4443 consensus_peers.cc:485] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:32.920586 4441 consensus_peers.cc:485] T 6c8f977981c54fa1bf5d173e83608146 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 20:46:32.923943 12396 raft_consensus.cc:482] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7845c397230d49358ba2fad9db53f97c)
I20240502 20:46:32.924064 12396 raft_consensus.cc:504] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: false } }
I20240502 20:46:32.924446 12396 leader_election.cc:288] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
W20240502 20:46:32.925319 4312 leader_election.cc:334] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:32.925413 4310 leader_election.cc:334] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
I20240502 20:46:32.925508 4310 leader_election.cc:302] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2; no voters: 201d2b6485084e7dad2cbb722af94779, 7845c397230d49358ba2fad9db53f97c
I20240502 20:46:32.925633 12396 raft_consensus.cc:2726] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:32.926669 4706 consensus_peers.cc:485] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:32.932821 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:33.038779 6768 consensus_peers.cc:485] T 52e302882311488ab1cac90ad03fe270 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:33.039815 4705 consensus_peers.cc:485] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:33.059026 4443 consensus_peers.cc:485] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:33.093653 4705 consensus_peers.cc:485] T 63f98288c8b1483493d5018c36e611bb P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:33.100509 6771 consensus_peers.cc:485] T a9afb75bfc114f59850eaea084a306f0 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:33.130484 4574 consensus_peers.cc:485] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:33.165515 6770 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111) [suppressed 151 similar messages]
W20240502 20:46:33.166273 6770 consensus_peers.cc:485] T 71537846373844cc9ef396b79af60db4 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:33.166457 4312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240502 20:46:33.171382 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:33.175983 4575 consensus_peers.cc:485] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:33.177086 4575 consensus_peers.cc:485] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:33.181191 4574 consensus_peers.cc:485] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:33.212599 4703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111) [suppressed 108 similar messages]
W20240502 20:46:33.214236 4441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111) [suppressed 125 similar messages]
W20240502 20:46:33.235873 4572 consensus_peers.cc:485] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:33.242020 4441 consensus_peers.cc:485] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:33.260278 4310 consensus_peers.cc:485] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 20:46:33.279563 12413 raft_consensus.cc:482] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7845c397230d49358ba2fad9db53f97c)
I20240502 20:46:33.279719 12413 raft_consensus.cc:504] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: true } }
I20240502 20:46:33.280184 12413 leader_election.cc:288] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
W20240502 20:46:33.281203 4572 leader_election.cc:334] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:33.281291 4574 leader_election.cc:334] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
I20240502 20:46:33.281356 4574 leader_election.cc:302] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91; no voters: 201d2b6485084e7dad2cbb722af94779, 7845c397230d49358ba2fad9db53f97c
I20240502 20:46:33.281494 12427 raft_consensus.cc:2726] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:33.307775 6771 consensus_peers.cc:485] T 6201083694c344c8ad63720be89a3cdb P a4bf1df7fa6b4948be502c182bfbb833 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:33.309451 4443 consensus_peers.cc:485] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:33.311386 4572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111) [suppressed 134 similar messages]
W20240502 20:46:33.312003 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
I20240502 20:46:33.315001 12420 raft_consensus.cc:482] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a47e5c56f13843018ac584914a8d4707)
I20240502 20:46:33.315105 12420 raft_consensus.cc:504] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:33.315452 12420 leader_election.cc:288] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
W20240502 20:46:33.316344 4444 leader_election.cc:334] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:33.316455 4441 leader_election.cc:334] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
I20240502 20:46:33.316529 4441 leader_election.cc:302] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3d26eafd51547acb780d4aac5ebe7cc; no voters: 201d2b6485084e7dad2cbb722af94779, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:33.316664 12420 raft_consensus.cc:2726] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:33.337782 4575 consensus_peers.cc:485] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 20:46:33.343686 12427 raft_consensus.cc:482] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7845c397230d49358ba2fad9db53f97c)
I20240502 20:46:33.343818 12427 raft_consensus.cc:504] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:33.344233 12427 leader_election.cc:288] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643), 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
W20240502 20:46:33.345341 4574 leader_election.cc:334] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:33.345422 4572 leader_election.cc:334] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
I20240502 20:46:33.345482 4572 leader_election.cc:302] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91; no voters: 201d2b6485084e7dad2cbb722af94779, 7845c397230d49358ba2fad9db53f97c
I20240502 20:46:33.345614 12427 raft_consensus.cc:2726] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:33.379061 4312 consensus_peers.cc:485] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:33.392202 4443 consensus_peers.cc:485] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:33.420529 6770 consensus_peers.cc:485] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:33.432289 4706 consensus_peers.cc:485] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:33.446151 6771 consensus_peers.cc:485] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:33.475119 4313 consensus_peers.cc:485] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:33.478554 4572 consensus_peers.cc:485] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:33.493450 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:33.512691 4444 consensus_peers.cc:485] T 28da7347f982468281570ffa7503a854 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 20:46:33.597520 12430 raft_consensus.cc:482] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:33.597651 12430 raft_consensus.cc:504] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:33.598019 12430 leader_election.cc:288] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
W20240502 20:46:33.598996 4703 leader_election.cc:334] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:33.599088 4706 leader_election.cc:334] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:33.599165 4706 leader_election.cc:302] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed47c023452a497192f35d2f617bf9bd; no voters: 201d2b6485084e7dad2cbb722af94779, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:33.599323 12430 raft_consensus.cc:2726] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 20:46:33.659350 6768 consensus_peers.cc:485] T d37730b109d04b6395f75c7552524877 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:33.684955 6770 consensus_peers.cc:485] T 7ca68870fae3480786c84bd86dc4897e P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:33.687284 4703 consensus_peers.cc:485] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:33.688938 4310 consensus_peers.cc:485] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:33.705578 4444 consensus_peers.cc:485] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:33.718156 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:33.727667 4574 consensus_peers.cc:485] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:33.749753 6771 consensus_peers.cc:485] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:33.803970 6768 consensus_peers.cc:485] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:33.848070 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:33.877763 6770 consensus_peers.cc:485] T dff789a7a50c4ff0a1eb29489ac7901a P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:34.058423 6768 consensus_peers.cc:485] T c60371165de94a5d9ec1c7181f839e5b P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 20:46:34.068205 12414 raft_consensus.cc:482] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a47e5c56f13843018ac584914a8d4707)
I20240502 20:46:34.068349 12414 raft_consensus.cc:504] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } }
I20240502 20:46:34.068755 12414 leader_election.cc:288] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
W20240502 20:46:34.069875 6770 leader_election.cc:334] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:34.069974 6771 leader_election.cc:334] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:34.070036 6771 leader_election.cc:302] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4bf1df7fa6b4948be502c182bfbb833; no voters: 7845c397230d49358ba2fad9db53f97c, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:34.070168 12414 raft_consensus.cc:2726] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:34.086285 4575 consensus_peers.cc:485] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 20:46:34.093700 12414 raft_consensus.cc:482] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7845c397230d49358ba2fad9db53f97c)
I20240502 20:46:34.093811 12414 raft_consensus.cc:504] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true } }
I20240502 20:46:34.094205 12414 leader_election.cc:288] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643), 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
W20240502 20:46:34.095021 6768 leader_election.cc:334] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:34.095129 6770 leader_election.cc:334] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
I20240502 20:46:34.095211 6770 leader_election.cc:302] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4bf1df7fa6b4948be502c182bfbb833; no voters: 201d2b6485084e7dad2cbb722af94779, 7845c397230d49358ba2fad9db53f97c
I20240502 20:46:34.095340 12414 raft_consensus.cc:2726] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:34.108464 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:34.191112 4575 consensus_peers.cc:485] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:34.242789 4703 consensus_peers.cc:485] T abe3bf6cb45a438fbd10bd5e78febc32 P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:34.254761 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:34.363364 4310 consensus_peers.cc:485] T a7ea68e8581a4cd5a6212abc96dfbcde P c92e6e46782849479c8a0c6d82276fb2 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:34.378067 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
I20240502 20:46:34.601497 12436 raft_consensus.cc:482] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a47e5c56f13843018ac584914a8d4707)
I20240502 20:46:34.601666 12436 raft_consensus.cc:504] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } }
I20240502 20:46:34.602162 12436 leader_election.cc:288] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
W20240502 20:46:34.602578 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:34.603260 4706 leader_election.cc:334] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:34.603343 4705 leader_election.cc:334] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
I20240502 20:46:34.603411 4705 leader_election.cc:302] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed47c023452a497192f35d2f617bf9bd; no voters: 7845c397230d49358ba2fad9db53f97c, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:34.603547 12436 raft_consensus.cc:2726] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:34.651257 4443 consensus_peers.cc:485] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:34.663861 4572 consensus_peers.cc:485] T 51232fccd98c461fb3d9016081f52ba5 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 20:46:34.774682 12413 raft_consensus.cc:482] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a47e5c56f13843018ac584914a8d4707)
I20240502 20:46:34.774833 12413 raft_consensus.cc:504] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true } }
I20240502 20:46:34.775286 12413 leader_election.cc:288] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
W20240502 20:46:34.776311 4574 leader_election.cc:334] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:34.776608 4575 leader_election.cc:334] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:34.776679 4575 leader_election.cc:302] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91; no voters: 7845c397230d49358ba2fad9db53f97c, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:34.776834 12439 raft_consensus.cc:2726] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:34.780922 4219 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240502 20:46:34.781735 4219 catalog_manager.cc:4573] TS 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): CreateTablet RPC failed for tablet 9ede0f8eef3440349ec5f0b93878c1f4: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:34.825314 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:34.868903 12396 raft_consensus.cc:482] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 201d2b6485084e7dad2cbb722af94779)
I20240502 20:46:34.869067 12396 raft_consensus.cc:504] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } }
I20240502 20:46:34.869467 12396 leader_election.cc:288] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
W20240502 20:46:34.870487 4313 leader_election.cc:334] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:34.870579 4310 leader_election.cc:334] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
I20240502 20:46:34.870671 4310 leader_election.cc:302] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2; no voters: 201d2b6485084e7dad2cbb722af94779, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:34.870826 12396 raft_consensus.cc:2726] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:34.961517 4312 consensus_peers.cc:485] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:34.969805 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
I20240502 20:46:35.017673 12396 raft_consensus.cc:482] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a47e5c56f13843018ac584914a8d4707)
I20240502 20:46:35.017805 12396 raft_consensus.cc:504] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:35.018203 12396 leader_election.cc:288] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
W20240502 20:46:35.019153 4313 leader_election.cc:334] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:35.025333 4310 leader_election.cc:334] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
I20240502 20:46:35.025514 4310 leader_election.cc:302] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2; no voters: 201d2b6485084e7dad2cbb722af94779, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:35.025736 12396 raft_consensus.cc:2726] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:35.030678 6768 consensus_peers.cc:485] T 6ed088a162a14d37b5da21ebb468a03b P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:35.047269 4444 consensus_peers.cc:485] T 747e7331d7ad4dd4b132672254037e1b P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:35.047889 4312 consensus_peers.cc:485] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:35.069361 4443 consensus_peers.cc:485] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:35.109826 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:35.114449 4706 consensus_peers.cc:485] T 9258667ccc3e4ef8b26fe75531dd4591 P ed47c023452a497192f35d2f617bf9bd -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:35.217883 6768 consensus_peers.cc:485] T 71537846373844cc9ef396b79af60db4 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:35.232659 4703 consensus_peers.cc:485] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:35.263660 4574 consensus_peers.cc:485] T 77dc51e6a85a4ddcb64db12b40c5810f P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:35.289085 4441 consensus_peers.cc:485] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:35.300416 4703 consensus_peers.cc:485] T 63f98288c8b1483493d5018c36e611bb P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:35.317940 6768 consensus_peers.cc:485] T 52e302882311488ab1cac90ad03fe270 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:35.334864 4703 consensus_peers.cc:485] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:35.359014 4705 consensus_peers.cc:485] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:35.373553 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:35.430527 4706 consensus_peers.cc:485] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:35.434589 4441 consensus_peers.cc:485] T 6c8f977981c54fa1bf5d173e83608146 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:35.472398 6768 consensus_peers.cc:485] T 6201083694c344c8ad63720be89a3cdb P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:35.510584 6770 consensus_peers.cc:485] T 71537846373844cc9ef396b79af60db4 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:35.517714 4574 consensus_peers.cc:485] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:35.534655 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:35.555629 4575 consensus_peers.cc:485] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:35.585964 4444 consensus_peers.cc:485] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:35.587853 6771 consensus_peers.cc:485] T a9afb75bfc114f59850eaea084a306f0 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:35.664402 4444 consensus_peers.cc:485] T 28da7347f982468281570ffa7503a854 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:35.669557 4443 consensus_peers.cc:485] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:35.684051 6770 consensus_peers.cc:485] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:35.689813 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:35.691262 4572 consensus_peers.cc:485] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:35.706013 4705 consensus_peers.cc:485] T 63f98288c8b1483493d5018c36e611bb P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:35.787755 4310 consensus_peers.cc:485] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:35.790977 4313 consensus_peers.cc:485] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:35.798534 6771 consensus_peers.cc:485] T 6201083694c344c8ad63720be89a3cdb P a4bf1df7fa6b4948be502c182bfbb833 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:35.814081 4441 consensus_peers.cc:485] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:35.823963 4443 consensus_peers.cc:485] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:35.843261 4574 consensus_peers.cc:485] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:35.847884 4443 consensus_peers.cc:485] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:35.861719 6771 consensus_peers.cc:485] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:35.884306 4706 consensus_peers.cc:485] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:35.914502 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:35.919831 4575 consensus_peers.cc:485] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:35.995848 4575 consensus_peers.cc:485] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:36.000057 6768 consensus_peers.cc:485] T d37730b109d04b6395f75c7552524877 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:36.037848 4574 consensus_peers.cc:485] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:36.046891 4312 consensus_peers.cc:485] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:36.054371 4572 consensus_peers.cc:485] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:36.071899 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:36.086614 4310 consensus_peers.cc:485] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:36.144670 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:36.192220 6771 consensus_peers.cc:485] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:36.226903 4444 consensus_peers.cc:485] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:36.308773 4703 consensus_peers.cc:485] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:36.319967 6768 consensus_peers.cc:485] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:36.321229 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:36.338551 6770 consensus_peers.cc:485] T dff789a7a50c4ff0a1eb29489ac7901a P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:36.366130 6770 consensus_peers.cc:485] T 7ca68870fae3480786c84bd86dc4897e P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:36.473145 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:36.527719 4575 consensus_peers.cc:485] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:36.558761 6768 consensus_peers.cc:485] T c60371165de94a5d9ec1c7181f839e5b P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:36.619102 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:36.711791 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:36.715612 4575 consensus_peers.cc:485] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:36.746592 4703 consensus_peers.cc:485] T abe3bf6cb45a438fbd10bd5e78febc32 P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:37.005406 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:37.007839 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:37.062906 4312 consensus_peers.cc:485] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:37.115635 4443 consensus_peers.cc:485] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:37.141592 4310 consensus_peers.cc:485] T a7ea68e8581a4cd5a6212abc96dfbcde P c92e6e46782849479c8a0c6d82276fb2 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:37.164820 4572 consensus_peers.cc:485] T 51232fccd98c461fb3d9016081f52ba5 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:37.263957 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:37.291975 4444 consensus_peers.cc:485] T 747e7331d7ad4dd4b132672254037e1b P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 20:46:37.315709 12396 raft_consensus.cc:482] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a47e5c56f13843018ac584914a8d4707)
I20240502 20:46:37.315858 12396 raft_consensus.cc:504] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:37.316318 12396 leader_election.cc:288] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
W20240502 20:46:37.317423 4313 leader_election.cc:334] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:37.317507 4312 leader_election.cc:334] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
I20240502 20:46:37.317561 4313 leader_election.cc:302] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2; no voters: 7845c397230d49358ba2fad9db53f97c, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:37.317760 12396 raft_consensus.cc:2726] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:37.452518 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:37.522886 4443 consensus_peers.cc:485] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:37.545852 4441 consensus_peers.cc:485] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:37.584059 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:37.623919 4706 consensus_peers.cc:485] T 9258667ccc3e4ef8b26fe75531dd4591 P ed47c023452a497192f35d2f617bf9bd -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:37.638358 4705 consensus_peers.cc:485] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:37.639304 4703 consensus_peers.cc:485] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:37.686744 6768 consensus_peers.cc:485] T 71537846373844cc9ef396b79af60db4 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:37.696722 4574 consensus_peers.cc:485] T 77dc51e6a85a4ddcb64db12b40c5810f P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:37.765285 4312 consensus_peers.cc:485] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:37.774024 6768 consensus_peers.cc:485] T 6ed088a162a14d37b5da21ebb468a03b P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:37.796523 4703 consensus_peers.cc:485] T 63f98288c8b1483493d5018c36e611bb P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:37.813930 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:37.819708 6770 consensus_peers.cc:485] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:37.836788 4706 consensus_peers.cc:485] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:37.842126 6768 consensus_peers.cc:485] T 6201083694c344c8ad63720be89a3cdb P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:37.892437 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:37.901080 6770 consensus_peers.cc:485] T 71537846373844cc9ef396b79af60db4 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:37.936931 4703 consensus_peers.cc:485] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:37.953863 4444 consensus_peers.cc:485] T 28da7347f982468281570ffa7503a854 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:37.973881 4574 consensus_peers.cc:485] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:37.982858 4441 consensus_peers.cc:485] T 6c8f977981c54fa1bf5d173e83608146 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:38.017385 6768 consensus_peers.cc:485] T 52e302882311488ab1cac90ad03fe270 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:38.046587 6771 consensus_peers.cc:485] T a9afb75bfc114f59850eaea084a306f0 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:38.117585 4572 consensus_peers.cc:485] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:38.127606 4443 consensus_peers.cc:485] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:38.130043 4443 consensus_peers.cc:485] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:38.177323 6771 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111) [suppressed 156 similar messages]
W20240502 20:46:38.187747 4310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240502 20:46:38.196843 4575 consensus_peers.cc:485] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:38.201962 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:38.205636 6768 consensus_peers.cc:485] T d37730b109d04b6395f75c7552524877 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:38.222193 6771 consensus_peers.cc:485] T 6201083694c344c8ad63720be89a3cdb P a4bf1df7fa6b4948be502c182bfbb833 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:38.222400 4706 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240502 20:46:38.255935 4310 consensus_peers.cc:485] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:38.261626 4444 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111) [suppressed 124 similar messages]
W20240502 20:46:38.262317 4444 consensus_peers.cc:485] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:38.280004 4313 consensus_peers.cc:485] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:38.316695 4706 consensus_peers.cc:485] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:38.318130 4574 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111) [suppressed 129 similar messages]
W20240502 20:46:38.323011 6771 consensus_peers.cc:485] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:38.375828 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
I20240502 20:46:38.378007 12458 raft_consensus.cc:482] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a47e5c56f13843018ac584914a8d4707)
I20240502 20:46:38.378126 12458 raft_consensus.cc:504] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true } }
I20240502 20:46:38.378484 12458 leader_election.cc:288] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
W20240502 20:46:38.379405 4575 leader_election.cc:334] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:38.379513 4574 leader_election.cc:334] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
I20240502 20:46:38.379592 4574 leader_election.cc:302] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91; no voters: 7845c397230d49358ba2fad9db53f97c, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:38.379727 12458 raft_consensus.cc:2726] T 8602022a45e1482ebf8dab9930ce8209 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:38.387133 4574 consensus_peers.cc:485] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:38.411907 4575 consensus_peers.cc:485] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:38.413059 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:38.467376 6770 consensus_peers.cc:485] T dff789a7a50c4ff0a1eb29489ac7901a P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:38.481740 4705 consensus_peers.cc:485] T 63f98288c8b1483493d5018c36e611bb P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:38.517241 4441 consensus_peers.cc:485] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:38.540189 4574 consensus_peers.cc:485] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:38.545601 4444 consensus_peers.cc:485] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:38.550433 4310 consensus_peers.cc:485] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:38.561378 4443 consensus_peers.cc:485] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:38.587792 4572 consensus_peers.cc:485] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:38.616197 4312 consensus_peers.cc:485] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:38.662490 4575 consensus_peers.cc:485] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:38.685786 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:38.688498 6771 consensus_peers.cc:485] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:38.742003 6770 consensus_peers.cc:485] T 7ca68870fae3480786c84bd86dc4897e P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:38.857853 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:38.930648 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:38.952030 4575 consensus_peers.cc:485] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:38.954147 6768 consensus_peers.cc:485] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:38.987715 6768 consensus_peers.cc:485] T c60371165de94a5d9ec1c7181f839e5b P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:39.016582 4703 consensus_peers.cc:485] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 20:46:39.222615 12456 raft_consensus.cc:482] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 201d2b6485084e7dad2cbb722af94779)
I20240502 20:46:39.222744 12456 raft_consensus.cc:504] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } attrs { replace: true } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } }
I20240502 20:46:39.223161 12456 leader_election.cc:288] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
W20240502 20:46:39.224507 6768 leader_election.cc:334] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:39.224774 6771 leader_election.cc:334] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:39.224843 6771 leader_election.cc:302] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4bf1df7fa6b4948be502c182bfbb833; no voters: 201d2b6485084e7dad2cbb722af94779, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:39.225009 12456 raft_consensus.cc:2726] T 2c67f965350a45f89511679d40f04cbb P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:39.227847 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:39.311898 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
I20240502 20:46:39.319180 12445 raft_consensus.cc:482] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7845c397230d49358ba2fad9db53f97c)
I20240502 20:46:39.319329 12445 raft_consensus.cc:504] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: NON_VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: true } }
I20240502 20:46:39.319722 12445 leader_election.cc:288] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
W20240502 20:46:39.320549 4574 leader_election.cc:334] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:39.320668 4572 leader_election.cc:334] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
I20240502 20:46:39.320725 4572 leader_election.cc:302] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91; no voters: 201d2b6485084e7dad2cbb722af94779, 7845c397230d49358ba2fad9db53f97c
I20240502 20:46:39.320839 12445 raft_consensus.cc:2726] T 3d938d01d5154590a9262acd54094ad2 P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:39.337457 4575 consensus_peers.cc:485] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:39.416348 4312 consensus_peers.cc:485] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:39.433245 4572 consensus_peers.cc:485] T 51232fccd98c461fb3d9016081f52ba5 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:39.450835 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:39.547114 4443 consensus_peers.cc:485] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:39.691387 4310 consensus_peers.cc:485] T a7ea68e8581a4cd5a6212abc96dfbcde P c92e6e46782849479c8a0c6d82276fb2 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:39.702028 4703 consensus_peers.cc:485] T abe3bf6cb45a438fbd10bd5e78febc32 P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:39.735036 4444 consensus_peers.cc:485] T 747e7331d7ad4dd4b132672254037e1b P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:39.782358 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:39.898324 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:39.937338 4703 consensus_peers.cc:485] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:39.957350 6768 consensus_peers.cc:485] T 71537846373844cc9ef396b79af60db4 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:40.001080 6768 consensus_peers.cc:485] T 6ed088a162a14d37b5da21ebb468a03b P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 20:46:40.043618 12466 raft_consensus.cc:482] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:40.043733 12466 raft_consensus.cc:504] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:40.044070 12466 leader_election.cc:288] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
W20240502 20:46:40.044852 4703 leader_election.cc:334] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:40.044966 4706 leader_election.cc:334] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:40.045056 4706 leader_election.cc:302] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed47c023452a497192f35d2f617bf9bd; no voters: 201d2b6485084e7dad2cbb722af94779, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:40.045176 12466 raft_consensus.cc:2726] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 20:46:40.053124 4706 consensus_peers.cc:485] T 9258667ccc3e4ef8b26fe75531dd4591 P ed47c023452a497192f35d2f617bf9bd -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:40.071725 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:40.076074 4312 consensus_peers.cc:485] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:40.111922 4441 consensus_peers.cc:485] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:40.209421 4443 consensus_peers.cc:485] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:40.220295 4705 consensus_peers.cc:485] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:40.262126 4703 consensus_peers.cc:485] T 63f98288c8b1483493d5018c36e611bb P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:40.286937 4574 consensus_peers.cc:485] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:40.319355 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:40.327889 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:40.331288 4574 consensus_peers.cc:485] T 77dc51e6a85a4ddcb64db12b40c5810f P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:40.371438 4444 consensus_peers.cc:485] T 28da7347f982468281570ffa7503a854 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:40.378965 6770 consensus_peers.cc:485] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:40.435945 6768 consensus_peers.cc:485] T 6201083694c344c8ad63720be89a3cdb P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:40.452000 4706 consensus_peers.cc:485] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:40.457538 4572 consensus_peers.cc:485] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:40.480062 6771 consensus_peers.cc:485] T a9afb75bfc114f59850eaea084a306f0 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:40.484519 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:40.502986 6770 consensus_peers.cc:485] T 71537846373844cc9ef396b79af60db4 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:40.520797 4443 consensus_peers.cc:485] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:40.579929 6768 consensus_peers.cc:485] T d37730b109d04b6395f75c7552524877 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:40.580397 4706 consensus_peers.cc:485] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:40.596467 4443 consensus_peers.cc:485] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:40.642542 6768 consensus_peers.cc:485] T 52e302882311488ab1cac90ad03fe270 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:40.648097 4441 consensus_peers.cc:485] T 6c8f977981c54fa1bf5d173e83608146 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:40.711758 4703 consensus_peers.cc:485] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:40.718233 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:40.754860 6771 consensus_peers.cc:485] T 6201083694c344c8ad63720be89a3cdb P a4bf1df7fa6b4948be502c182bfbb833 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:40.872318 4444 consensus_peers.cc:485] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:40.873446 4313 consensus_peers.cc:485] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:40.875893 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:40.893787 4312 consensus_peers.cc:485] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:40.900002 4575 consensus_peers.cc:485] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:40.903015 4574 consensus_peers.cc:485] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:40.927487 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:40.960114 4575 consensus_peers.cc:485] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:40.998854 4310 consensus_peers.cc:485] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:41.014389 4310 consensus_peers.cc:485] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:41.028216 6771 consensus_peers.cc:485] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:41.031006 4705 consensus_peers.cc:485] T 63f98288c8b1483493d5018c36e611bb P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 20:46:41.054949 12396 raft_consensus.cc:482] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7845c397230d49358ba2fad9db53f97c)
I20240502 20:46:41.055096 12396 raft_consensus.cc:504] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: false } }
I20240502 20:46:41.055538 12396 leader_election.cc:288] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
W20240502 20:46:41.055732 4441 consensus_peers.cc:485] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:41.056463 4312 leader_election.cc:334] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:41.056555 4310 leader_election.cc:334] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
I20240502 20:46:41.056617 4310 leader_election.cc:302] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2; no voters: 201d2b6485084e7dad2cbb722af94779, 7845c397230d49358ba2fad9db53f97c
I20240502 20:46:41.056756 12396 raft_consensus.cc:2726] T ac57ad308b174217ac74c4cb5e1c7285 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:41.095553 4444 consensus_peers.cc:485] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:41.113310 6771 consensus_peers.cc:485] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:41.113729 6770 consensus_peers.cc:485] T dff789a7a50c4ff0a1eb29489ac7901a P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:41.131932 4574 consensus_peers.cc:485] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:41.143198 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:41.179316 6768 consensus_peers.cc:485] T c60371165de94a5d9ec1c7181f839e5b P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:41.200644 4703 consensus_peers.cc:485] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:41.205740 4572 consensus_peers.cc:485] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:41.229861 4443 consensus_peers.cc:485] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:41.294667 4575 consensus_peers.cc:485] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:41.316802 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:41.325878 6770 consensus_peers.cc:485] T 7ca68870fae3480786c84bd86dc4897e P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 20:46:41.335528 12396 raft_consensus.cc:482] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a47e5c56f13843018ac584914a8d4707)
I20240502 20:46:41.335659 12396 raft_consensus.cc:504] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:41.336073 12396 leader_election.cc:288] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
W20240502 20:46:41.337060 4313 leader_election.cc:334] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:41.337142 4312 leader_election.cc:334] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
I20240502 20:46:41.337205 4312 leader_election.cc:302] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2; no voters: 7845c397230d49358ba2fad9db53f97c, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:41.337342 12396 raft_consensus.cc:2726] T 681478951a8143f9959b78eec15efaf1 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:41.372898 6768 consensus_peers.cc:485] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:41.419486 4575 consensus_peers.cc:485] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:41.481452 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
I20240502 20:46:41.492854 12445 raft_consensus.cc:482] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7845c397230d49358ba2fad9db53f97c)
I20240502 20:46:41.493009 12445 raft_consensus.cc:504] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } }
I20240502 20:46:41.493556 12445 leader_election.cc:288] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643), 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
W20240502 20:46:41.494393 4572 leader_election.cc:334] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:41.494513 4574 leader_election.cc:334] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
I20240502 20:46:41.494587 4574 leader_election.cc:302] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91; no voters: 201d2b6485084e7dad2cbb722af94779, 7845c397230d49358ba2fad9db53f97c
I20240502 20:46:41.494769 12445 raft_consensus.cc:2726] T cfe54b00aa2a4dbe98440c9320a3deef P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:41.531834 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:41.775142 4312 consensus_peers.cc:485] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:41.796595 4443 consensus_peers.cc:485] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:41.819969 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:41.898152 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:41.898190 4575 consensus_peers.cc:485] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 20:46:42.006633 12482 raft_consensus.cc:482] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7845c397230d49358ba2fad9db53f97c)
I20240502 20:46:42.006763 12482 raft_consensus.cc:504] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "923cfab926e7443f8ffd7b21ad0a9d91" member_type: VOTER last_known_addr { host: "127.30.32.67" port: 42141 } attrs { promote: false } }
I20240502 20:46:42.007086 12482 leader_election.cc:288] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
W20240502 20:46:42.008004 4574 leader_election.cc:334] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:42.008114 4572 leader_election.cc:334] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
I20240502 20:46:42.008196 4572 leader_election.cc:302] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 923cfab926e7443f8ffd7b21ad0a9d91; no voters: 201d2b6485084e7dad2cbb722af94779, 7845c397230d49358ba2fad9db53f97c
I20240502 20:46:42.008332 12482 raft_consensus.cc:2726] T a135d04d8c3248bc954a07e5b6396fde P 923cfab926e7443f8ffd7b21ad0a9d91 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:42.032760 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:42.085826 4572 consensus_peers.cc:485] T 51232fccd98c461fb3d9016081f52ba5 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:42.117368 4703 consensus_peers.cc:485] T abe3bf6cb45a438fbd10bd5e78febc32 P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:42.204389 4444 consensus_peers.cc:485] T 747e7331d7ad4dd4b132672254037e1b P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:42.239646 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:42.355409 6768 consensus_peers.cc:485] T 71537846373844cc9ef396b79af60db4 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:42.367434 4310 consensus_peers.cc:485] T a7ea68e8581a4cd5a6212abc96dfbcde P c92e6e46782849479c8a0c6d82276fb2 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:42.389384 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:42.441946 4312 consensus_peers.cc:485] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:42.472704 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:42.527413 4705 consensus_peers.cc:485] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:42.593956 4703 consensus_peers.cc:485] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:42.605757 4706 consensus_peers.cc:485] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:42.610157 4706 consensus_peers.cc:485] T 9258667ccc3e4ef8b26fe75531dd4591 P ed47c023452a497192f35d2f617bf9bd -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:42.619725 4441 consensus_peers.cc:485] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:42.669543 6768 consensus_peers.cc:485] T 6ed088a162a14d37b5da21ebb468a03b P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:42.684748 4703 consensus_peers.cc:485] T 63f98288c8b1483493d5018c36e611bb P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:42.768740 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:42.772338 6770 consensus_peers.cc:485] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:42.793623 4443 consensus_peers.cc:485] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:42.805022 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:42.852411 4574 consensus_peers.cc:485] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:42.876233 4706 consensus_peers.cc:485] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:42.878984 4444 consensus_peers.cc:485] T 28da7347f982468281570ffa7503a854 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:42.922461 4574 consensus_peers.cc:485] T 77dc51e6a85a4ddcb64db12b40c5810f P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:42.963848 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:42.968386 4443 consensus_peers.cc:485] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:42.971438 6768 consensus_peers.cc:485] T 6201083694c344c8ad63720be89a3cdb P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:42.975476 6770 consensus_peers.cc:485] T 71537846373844cc9ef396b79af60db4 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:43.048115 4441 consensus_peers.cc:485] T 6c8f977981c54fa1bf5d173e83608146 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 20:46:43.067695 12489 raft_consensus.cc:482] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7845c397230d49358ba2fad9db53f97c)
I20240502 20:46:43.067835 12489 raft_consensus.cc:504] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } }
I20240502 20:46:43.068225 12489 leader_election.cc:288] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
W20240502 20:46:43.069150 4313 leader_election.cc:334] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:43.069305 4312 leader_election.cc:334] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
I20240502 20:46:43.069379 4312 leader_election.cc:302] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2; no voters: 7845c397230d49358ba2fad9db53f97c, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:43.069506 12489 raft_consensus.cc:2726] T 5ed65a0bf8c0469eb6ea91bcf4f38daa P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:43.096964 4443 consensus_peers.cc:485] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:43.099076 4703 consensus_peers.cc:485] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:43.109730 6771 consensus_peers.cc:485] T 6201083694c344c8ad63720be89a3cdb P a4bf1df7fa6b4948be502c182bfbb833 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:43.158986 4572 consensus_peers.cc:485] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:43.187884 6771 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111) [suppressed 152 similar messages]
W20240502 20:46:43.188575 6771 consensus_peers.cc:485] T a9afb75bfc114f59850eaea084a306f0 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:43.241685 6768 consensus_peers.cc:485] T 52e302882311488ab1cac90ad03fe270 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:43.255049 4313 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111) [suppressed 85 similar messages]
W20240502 20:46:43.255668 4313 consensus_peers.cc:485] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:43.258049 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:43.278112 4312 consensus_peers.cc:485] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:43.278553 4444 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240502 20:46:43.279178 4444 consensus_peers.cc:485] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:43.287600 4706 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111) [suppressed 117 similar messages]
W20240502 20:46:43.320256 4444 consensus_peers.cc:485] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:43.329658 6768 consensus_peers.cc:485] T d37730b109d04b6395f75c7552524877 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:43.368959 4572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111) [suppressed 136 similar messages]
W20240502 20:46:43.369643 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:43.369764 4575 consensus_peers.cc:485] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:43.409240 6768 consensus_peers.cc:485] T c60371165de94a5d9ec1c7181f839e5b P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:43.434594 4575 consensus_peers.cc:485] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:43.444861 4310 consensus_peers.cc:485] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:43.478550 4705 consensus_peers.cc:485] T 63f98288c8b1483493d5018c36e611bb P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:43.482034 4441 consensus_peers.cc:485] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:43.494562 6771 consensus_peers.cc:485] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:43.525883 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:43.539201 4310 consensus_peers.cc:485] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:43.569535 4703 consensus_peers.cc:485] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:43.582072 6771 consensus_peers.cc:485] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:43.591817 4574 consensus_peers.cc:485] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:43.616194 4574 consensus_peers.cc:485] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 20:46:43.631834 12484 raft_consensus.cc:482] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a47e5c56f13843018ac584914a8d4707)
I20240502 20:46:43.631966 12484 raft_consensus.cc:504] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: NON_VOTER last_known_addr { host: "127.30.32.66" port: 42097 } attrs { promote: true } }
I20240502 20:46:43.632315 12484 leader_election.cc:288] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
W20240502 20:46:43.633275 6770 leader_election.cc:334] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:43.633503 6771 leader_election.cc:334] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:43.633563 6771 leader_election.cc:302] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4bf1df7fa6b4948be502c182bfbb833; no voters: 7845c397230d49358ba2fad9db53f97c, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:43.633785 12484 raft_consensus.cc:2726] T a999f994b78a4b33a95ee7d5407cba20 P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:43.660332 6770 consensus_peers.cc:485] T 7ca68870fae3480786c84bd86dc4897e P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:43.714902 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:43.739130 6770 consensus_peers.cc:485] T dff789a7a50c4ff0a1eb29489ac7901a P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:43.749349 4572 consensus_peers.cc:485] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:43.868805 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:43.931615 4443 consensus_peers.cc:485] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:43.947085 6768 consensus_peers.cc:485] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:44.016954 4575 consensus_peers.cc:485] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:44.086598 4575 consensus_peers.cc:485] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:44.118276 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:44.296674 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:44.303256 4312 consensus_peers.cc:485] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:46:44.326488 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:44.464218 4443 consensus_peers.cc:485] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:44.490926 4575 consensus_peers.cc:485] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 20:46:44.534933 12489 raft_consensus.cc:482] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 201d2b6485084e7dad2cbb722af94779)
I20240502 20:46:44.535066 12489 raft_consensus.cc:504] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: NON_VOTER last_known_addr { host: "127.30.32.69" port: 42429 } attrs { promote: true } }
I20240502 20:46:44.535444 12489 leader_election.cc:288] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
W20240502 20:46:44.536413 4313 leader_election.cc:334] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:44.536557 4310 leader_election.cc:334] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
I20240502 20:46:44.536618 4310 leader_election.cc:302] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2; no voters: 201d2b6485084e7dad2cbb722af94779, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:44.536783 12489 raft_consensus.cc:2726] T d78c02773e454ea0b79cf1282eda125b P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 20:46:44.635494 12500 raft_consensus.cc:482] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a47e5c56f13843018ac584914a8d4707)
I20240502 20:46:44.635630 12500 raft_consensus.cc:504] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d26eafd51547acb780d4aac5ebe7cc" member_type: VOTER last_known_addr { host: "127.30.32.66" port: 42097 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:44.636013 12500 leader_election.cc:288] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
W20240502 20:46:44.636932 4441 leader_election.cc:334] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:44.637053 4444 leader_election.cc:334] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:44.637136 4444 leader_election.cc:302] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3d26eafd51547acb780d4aac5ebe7cc; no voters: 201d2b6485084e7dad2cbb722af94779, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:44.637282 12500 raft_consensus.cc:2726] T 935f0efc1fe146f4ac21fac4c0cc688c P c3d26eafd51547acb780d4aac5ebe7cc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 20:46:44.661576 12498 raft_consensus.cc:482] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7845c397230d49358ba2fad9db53f97c)
I20240502 20:46:44.661746 12498 raft_consensus.cc:504] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } attrs { replace: true } } peers { permanent_uuid: "a4bf1df7fa6b4948be502c182bfbb833" member_type: VOTER last_known_addr { host: "127.30.32.70" port: 41867 } } peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: NON_VOTER last_known_addr { host: "127.30.32.65" port: 33827 } attrs { promote: true } }
I20240502 20:46:44.662199 12498 leader_election.cc:288] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643), 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429)
W20240502 20:46:44.663115 6768 leader_election.cc:334] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:44.663228 6770 leader_election.cc:334] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
I20240502 20:46:44.663291 6770 leader_election.cc:302] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4bf1df7fa6b4948be502c182bfbb833; no voters: 201d2b6485084e7dad2cbb722af94779, 7845c397230d49358ba2fad9db53f97c
I20240502 20:46:44.663434 12498 raft_consensus.cc:2726] T 6ff3c95ccfbf486fbc4bfc636afa829b P a4bf1df7fa6b4948be502c182bfbb833 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:44.680895 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:44.687904 4572 consensus_peers.cc:485] T 51232fccd98c461fb3d9016081f52ba5 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:44.699323 4703 consensus_peers.cc:485] T abe3bf6cb45a438fbd10bd5e78febc32 P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:44.719517 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:44.730845 6768 consensus_peers.cc:485] T 71537846373844cc9ef396b79af60db4 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:46:44.757006 4444 consensus_peers.cc:485] T 747e7331d7ad4dd4b132672254037e1b P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:44.815029 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:44.920385 4441 consensus_peers.cc:485] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:46:44.990170 4310 consensus_peers.cc:485] T a7ea68e8581a4cd5a6212abc96dfbcde P c92e6e46782849479c8a0c6d82276fb2 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 20:46:45.010279 12466 raft_consensus.cc:482] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:45.010411 12466 raft_consensus.cc:504] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } }
I20240502 20:46:45.010881 12466 leader_election.cc:288] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
W20240502 20:46:45.011930 4703 leader_election.cc:334] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:45.012003 4706 leader_election.cc:334] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:45.012045 4703 leader_election.cc:302] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed47c023452a497192f35d2f617bf9bd; no voters: 201d2b6485084e7dad2cbb722af94779, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:45.012226 12466 raft_consensus.cc:2726] T 9ede0f8eef3440349ec5f0b93878c1f4 P ed47c023452a497192f35d2f617bf9bd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 20:46:45.029603 4703 consensus_peers.cc:485] T 63f98288c8b1483493d5018c36e611bb P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:46:45.052345 4705 consensus_peers.cc:485] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 20:46:45.065474 12466 raft_consensus.cc:482] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a47e5c56f13843018ac584914a8d4707)
I20240502 20:46:45.065582 12466 raft_consensus.cc:504] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7845c397230d49358ba2fad9db53f97c" member_type: VOTER last_known_addr { host: "127.30.32.69" port: 42429 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "ed47c023452a497192f35d2f617bf9bd" member_type: VOTER last_known_addr { host: "127.30.32.68" port: 38193 } attrs { promote: false } }
I20240502 20:46:45.065941 12466 leader_election.cc:288] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429), a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841)
W20240502 20:46:45.066807 4705 leader_election.cc:334] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:45.066900 4706 leader_election.cc:334] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:45.066979 4706 leader_election.cc:302] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed47c023452a497192f35d2f617bf9bd; no voters: 7845c397230d49358ba2fad9db53f97c, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:45.067104 12466 raft_consensus.cc:2726] T bcfd626ea0a04fa7b03dd0ede6b35266 P ed47c023452a497192f35d2f617bf9bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:45.075330 4706 consensus_peers.cc:485] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:45.094419 4706 consensus_peers.cc:485] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:46:45.120661 4706 consensus_peers.cc:485] T 9258667ccc3e4ef8b26fe75531dd4591 P ed47c023452a497192f35d2f617bf9bd -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:45.145555 6770 consensus_peers.cc:485] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:46:45.154865 4443 consensus_peers.cc:485] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:46:45.192071 4312 consensus_peers.cc:485] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 20:46:45.201014 12489 raft_consensus.cc:482] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a47e5c56f13843018ac584914a8d4707)
I20240502 20:46:45.201166 12489 raft_consensus.cc:504] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92e6e46782849479c8a0c6d82276fb2" member_type: VOTER last_known_addr { host: "127.30.32.65" port: 33827 } } peers { permanent_uuid: "a47e5c56f13843018ac584914a8d4707" member_type: VOTER last_known_addr { host: "127.30.32.71" port: 46841 } } peers { permanent_uuid: "201d2b6485084e7dad2cbb722af94779" member_type: VOTER last_known_addr { host: "127.30.32.72" port: 44643 } }
I20240502 20:46:45.201663 12489 leader_election.cc:288] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841), 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643)
W20240502 20:46:45.202724 4310 leader_election.cc:334] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:45.202841 4313 leader_election.cc:334] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
I20240502 20:46:45.202936 4313 leader_election.cc:302] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c92e6e46782849479c8a0c6d82276fb2; no voters: 201d2b6485084e7dad2cbb722af94779, a47e5c56f13843018ac584914a8d4707
I20240502 20:46:45.203083 12489 raft_consensus.cc:2726] T 8bbf33f0ec4a4adb9a979b08606ee822 P c92e6e46782849479c8a0c6d82276fb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:45.224745 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:45.231170 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:45.304709 4574 consensus_peers.cc:485] T 77dc51e6a85a4ddcb64db12b40c5810f P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:45.338521 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:45.379298 4444 consensus_peers.cc:485] T 28da7347f982468281570ffa7503a854 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:46:45.420691 4703 consensus_peers.cc:485] T 20373b53281a4214b27f014396f7a651 P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:46:45.458343 4441 consensus_peers.cc:485] T 6c8f977981c54fa1bf5d173e83608146 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:46:45.499137 4443 consensus_peers.cc:485] T a62a7ce526f14c25994bb9e0e3c3bc84 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:46:45.507519 6768 consensus_peers.cc:485] T 6ed088a162a14d37b5da21ebb468a03b P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:45.521939 4443 consensus_peers.cc:485] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:46:45.559298 6770 consensus_peers.cc:485] T 71537846373844cc9ef396b79af60db4 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:46:45.567714 4574 consensus_peers.cc:485] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:46:45.575639 6768 consensus_peers.cc:485] T 6201083694c344c8ad63720be89a3cdb P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:46:45.624109 4312 consensus_peers.cc:485] T 93b7f7de1f8c49389c3d8c1e03c67f26 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:46:45.645242 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:45.663610 6768 consensus_peers.cc:485] T 52e302882311488ab1cac90ad03fe270 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:46:45.697428 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:45.759809 4572 consensus_peers.cc:485] T eebddad3efe748f78372901ecad5b407 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:46:45.769412 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:45.786661 4313 consensus_peers.cc:485] T c55ba788f592448d8941ff94a8cefc52 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:46:45.786773 4444 consensus_peers.cc:485] T 45ade8c672d8445eaffb179772e3759a P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:46:45.795866 4444 consensus_peers.cc:485] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:46:45.802351 6771 consensus_peers.cc:485] T a9afb75bfc114f59850eaea084a306f0 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:46:45.814065 6768 consensus_peers.cc:485] T d37730b109d04b6395f75c7552524877 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:46:45.817649 4705 consensus_peers.cc:485] T 63f98288c8b1483493d5018c36e611bb P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:46:45.824594 4703 consensus_peers.cc:485] T cdf218d5e45b4db19a8e2c22dbc5cc5d P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:46:45.847834 6771 consensus_peers.cc:485] T 6201083694c344c8ad63720be89a3cdb P a4bf1df7fa6b4948be502c182bfbb833 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:46:45.907318 4310 consensus_peers.cc:485] T 8e7e145762bf458a8374bfc503025a17 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:46:45.958854 6771 consensus_peers.cc:485] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:46:45.965302 6768 consensus_peers.cc:485] T c60371165de94a5d9ec1c7181f839e5b P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:46:46.000713 6771 consensus_peers.cc:485] T 78d87e2d8c0b4bbd8dd6d7f49cdb4287 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:46:46.000778 4575 consensus_peers.cc:485] T d37fab152a8a44868d3160ec3057bb98 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:46:46.026520 4441 consensus_peers.cc:485] T d6fb394fdcff436886fe0634d79c024c P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:46:46.045912 4574 consensus_peers.cc:485] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:46:46.083477 4574 consensus_peers.cc:485] T 02302fc79dc04b0396041c600c2f4bef P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:46:46.090229 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:46.095710 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:46.120815 4575 consensus_peers.cc:485] T 0cf7e95cb91b4e74b14365fcc6b7f968 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:46:46.152787 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:46.180858 4703 consensus_peers.cc:485] T 77b294bcc497479e8e8bd8e7f9cc1ff2 P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:46:46.308593 4572 consensus_peers.cc:485] T 76b6dee18d224e879bf0d3b6bbb2b9b5 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:46:46.312927 4310 consensus_peers.cc:485] T 15a2cbe8285a4938be7f628f7f8b734d P c92e6e46782849479c8a0c6d82276fb2 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:46:46.371371 6770 consensus_peers.cc:485] T dff789a7a50c4ff0a1eb29489ac7901a P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:46:46.436350 4575 consensus_peers.cc:485] T 00eb91e624554baf918e96b098019be8 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:46:46.464676 6770 consensus_peers.cc:485] T 7ca68870fae3480786c84bd86dc4897e P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:46:46.479198 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:46.494901 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:46.549875 4443 consensus_peers.cc:485] T 8565ebbd21dd4f708a0254d1ed2d6670 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:46:46.570223 6768 consensus_peers.cc:485] T b2508a5aa1c14b63971895953e149e17 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:46:46.613049 4575 consensus_peers.cc:485] T 7abfd61f55554042978b50761cbd6ab5 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:46:46.636886 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:46.840893 4443 consensus_peers.cc:485] T 75e2e48cd7a4497f9ccaae4f712f3ef0 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:46:46.971825 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:46.995319 4312 consensus_peers.cc:485] T c9d25657e9a14b35b2f6e8d82f4533c7 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:46:46.999143 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:47.156646 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
W20240502 20:46:47.163817 4572 consensus_peers.cc:485] T 51232fccd98c461fb3d9016081f52ba5 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:46:47.217273 4575 consensus_peers.cc:485] T 428ffbac75db466191cb3bef63b4eb79 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:46:47.254083 4444 consensus_peers.cc:485] T 747e7331d7ad4dd4b132672254037e1b P c3d26eafd51547acb780d4aac5ebe7cc -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:46:47.292752 4703 consensus_peers.cc:485] T abe3bf6cb45a438fbd10bd5e78febc32 P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:46:47.331041 6768 consensus_peers.cc:485] T 71537846373844cc9ef396b79af60db4 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:46:47.369966 4441 consensus_peers.cc:485] T 41f84d5db2034af591b56e109be67934 P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:46:47.383071 4312 consensus_peers.cc:485] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:46:47.443298 4705 consensus_peers.cc:485] T a2e81d2c63cd4e4cac5086506b5c4a12 P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:46:47.445417 4705 consensus_peers.cc:462] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111)
W20240502 20:46:47.454803 4572 consensus_peers.cc:462] T 253d158b340a444182793f18239bfa22 P 923cfab926e7443f8ffd7b21ad0a9d91 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111)
W20240502 20:46:47.480834 4706 consensus_peers.cc:485] T ef4002126e8847809add182f262d62d9 P ed47c023452a497192f35d2f617bf9bd -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:46:47.549217 4310 consensus_peers.cc:485] T a7ea68e8581a4cd5a6212abc96dfbcde P c92e6e46782849479c8a0c6d82276fb2 -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:46:47.598990 4706 consensus_peers.cc:485] T 73efeba3d3094ef387e4efb03cff9d4e P ed47c023452a497192f35d2f617bf9bd -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:46:47.685446 4313 consensus_peers.cc:462] T cedd1c79256a4f32a73b6a88244e6181 P c92e6e46782849479c8a0c6d82276fb2 -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111)
F20240502 20:46:47.699365 7148 rebalancer_tool-test.cc:1068] Check failed: _s.ok() Bad status: Timed out: Timed out waiting for Table Creation
*** Check failure stack trace: ***
*** Aborted at 1714682807 (unix time) try "date -d @1714682807" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800007881) received by PID 30849 (TID 0x7fe0862cb700) from PID 30849; stack trace: ***
@ 0x7fe09b2b5980 (unknown) at ??:0
@ 0x7fe09888bfb7 gsignal at ??:0
@ 0x7fe09888d921 abort at ??:0
@ 0x7fe099acbdcd google::LogMessage::Fail() at ??:0
@ 0x7fe099acfb93 google::LogMessage::SendToLog() at ??:0
W20240502 20:46:47.714560 4443 consensus_peers.cc:485] T d6582234196c4fe4a82b3182787700de P c3d26eafd51547acb780d4aac5ebe7cc -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
@ 0x7fe099acb7cc google::LogMessage::Flush() at ??:0
@ 0x7fe099accf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x5609fc1144cd _ZZN4kudu5tools70DDLDuringRebalancingTest_TablesCreatedAndDeletedDuringRebalancing_Test8TestBodyEvENKUlvE_clEv at ??:0
@ 0x5609fc12e216 _ZSt13__invoke_implIvZN4kudu5tools70DDLDuringRebalancingTest_TablesCreatedAndDeletedDuringRebalancing_Test8TestBodyEvEUlvE_JEET_St14__invoke_otherOT0_DpOT1_ at ??:0
@ 0x5609fc12d006 _ZSt8__invokeIZN4kudu5tools70DDLDuringRebalancingTest_TablesCreatedAndDeletedDuringRebalancing_Test8TestBodyEvEUlvE_JEENSt15__invoke_resultIT_JDpT0_EE4typeEOS5_DpOS6_ at ??:0
@ 0x5609fc134d34 _ZNSt6thread8_InvokerISt5tupleIJZN4kudu5tools70DDLDuringRebalancingTest_TablesCreatedAndDeletedDuringRebalancing_Test8TestBodyEvEUlvE_EEE9_M_invokeIJLm0EEEEDTcl8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at ??:0
@ 0x5609fc134bad _ZNSt6thread8_InvokerISt5tupleIJZN4kudu5tools70DDLDuringRebalancingTest_TablesCreatedAndDeletedDuringRebalancing_Test8TestBodyEvEUlvE_EEEclEv at ??:0
@ 0x5609fc134a70 _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN4kudu5tools70DDLDuringRebalancingTest_TablesCreatedAndDeletedDuringRebalancing_Test8TestBodyEvEUlvE_EEEEE6_M_runEv at ??:0
@ 0x7fe0992b16df (unknown) at ??:0
@ 0x7fe09b2aa6db start_thread at ??:0
@ 0x7fe09896e71f clone at ??:0
W20240502 20:46:47.779105 6770 consensus_peers.cc:485] T 180902058815460189f75de4704d1e84 P a4bf1df7fa6b4948be502c182bfbb833 -> Peer 7845c397230d49358ba2fad9db53f97c (127.30.32.69:42429): Couldn't send request to peer 7845c397230d49358ba2fad9db53f97c. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.69:42429: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:46:47.825516 4703 consensus_peers.cc:485] T 63f98288c8b1483493d5018c36e611bb P ed47c023452a497192f35d2f617bf9bd -> Peer 201d2b6485084e7dad2cbb722af94779 (127.30.32.72:44643): Couldn't send request to peer 201d2b6485084e7dad2cbb722af94779. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.72:44643: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:46:47.840718 4706 consensus_peers.cc:485] T 9258667ccc3e4ef8b26fe75531dd4591 P ed47c023452a497192f35d2f617bf9bd -> Peer a47e5c56f13843018ac584914a8d4707 (127.30.32.71:46841): Couldn't send request to peer a47e5c56f13843018ac584914a8d4707. Status: Network error: Client connection negotiation failed: client connection to 127.30.32.71:46841: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.